Processing resource management in an island-based network flow processor

ABSTRACT

An island-based network flow processor (IB-NFP) integrated circuit has a high performance processor island. The processor island has a processor and a tightly coupled memory. The integrated circuit also has another memory. The other memory may be internal or external memory. The header of an incoming packet is stored in the tightly coupled memory of the processor island. The payload is stored in the other memory. In one example, if the amount of a processing resource is below a threshold then the header is moved from the first island to the other memory before the header and payload are communicated to an egress island for outputting from the integrated circuit. If, however, the amount of the processing resource is not below the threshold then the header is moved directly from the processor island to the egress island and is combined with the payload there for outputting from the integrated circuit.

TECHNICAL FIELD

The described embodiments relate generally to island-based integratedcircuits and related methods.

BACKGROUND INFORMATION

A network processor is a device that executes programs to handle packettraffic in a data network. Examples include network processors on routerline cards and in other network equipment. The Intel IXP2800 is amulti-threaded multiprocessor integrated circuit that is capable ofreceiving packets, classifying and performing atomic operations on thepackets and packet data, and transmitting packets. Within the IXP2800integrated circuit, microengines operate on packet data stored in amemory. The memory is accessible to the microengines via a DRAMcommand/push/pull bus. The IXP2800 architecture is therefore flexibleand expandable in that more processing power can be added and coupled tothe bus so that the added processors will have access to the memory. Afamily of network processors of this architecture can be made byproviding network processor integrated circuits with different amountsof processing power and with different amounts of memory.

SUMMARY

In a first novel aspect, an island-based network flow processor (IB-NFP)integrated circuit has a configurable mesh data bus. The IB-NFPintegrated circuit has functional circuitry including ingress packetclassification circuits, processors, memories, and egress schedulingcircuitry. The functional circuitry is partitioned into rectangularislands. In addition to the functional circuitry, each island has a partof a mesh data bus. The part of the mesh data bus includes a crossbarswitch located centrally in the island, and a set of half links. Themesh data bus may for a sort of overlay over the functional circuitry ofthe islands. The islands are organized such that half links of adjacentislands are substantially collinear and join together to form links thatextend in substantially straight lines between crossbar switches ofadjacent islands.

In one specific example, there are four substantially identical meshbuses that together form a configurable command/push/pull data bus. Afirst mesh bus is a command mesh bus of the configurable mesh data bus,a second mesh bus is a pull-id mesh bus of the configurable mesh databus, a third mesh bus is a data1 mesh bus of the configurable mesh databus, and a fourth mesh bus is a data0 mesh bus of the configurable meshdata bus. Each crossbar switch of such a mesh data bus includes aplurality of Look Up Table (LUT) memories, the contents of whichconfigure the configurable mesh bus. In one example, a plurality ofcommands can be communicated simultaneously across different parts ofthe configurable mesh data bus. The configurable mesh data bus can beconfigured such that multiple different functional circuits in multipledifferent islands can be performing multiple reads or writes across theconfigurable mesh data bus at the same time.

In a second novel aspect, the islands of an island-based network flowprocessor (IB-NFP) integrated circuit are organized in a staggeredfashion. The islands are rectangular and are organized in rows. The leftand rights side edges of the islands of one row are offset laterally andare laterally staggered with respect to the left and right side edges ofthe islands of the next row. This lateral staggering, back and forth,from row to row, is such that the half links of the islands meet at theport locations and form a mesh bus structure. The left and right edgesof islands in a row align with left and right edges of islands two rowsdown in the row structure. In one specific example, two of the halflinks extend from the crossbar switch up to port locations on an topedge of the island, another of the half links extends from the crossbarswitch to the right to the right side edge of the island, two others ofthe half links extend down from the crossbar switch to port locations ona bottom edge of the island, another of the half links extends from thecentral location to the left to the left edge of the island. Two otherlinks extend from the crossbar switch to functional circuitry in theisland. In this way, six of the half links radiate outwardly from thecentral location of the crossbar switch and have a star-shape when theisland structure is considered from a top-down perspective. The halflinks and islands are oriented such that half links of adjacent islandsjoin together to form a mesh bus structure. There are four such mesh busstructures. One for the command bus of a command/push/pull data bus,another for the pull-id bus of the command/push/pull data bus, and twofor data buses of the command/push/pull data bus.

In a third novel aspect, each link of an island-based network flowprocessor (IB-NFP) integrated circuit is realized as a distributedcredit First-In-First-Out (FIFO) structure. The distributed credit FIFOstructure communicates information from the center part of a firstisland, through an output port of the first island, through an inputport of a second island, and to a center part of the second island. Thedistributed credit FIFO structure includes a first FIFO associated withan output port of a first island, a chain of registers, a second FIFOassociated with an input port of a second island, a second chain ofregisters, and a credit count circuit. A data path extends from thefirst FIFO in the first island, through the chain of registers, and tothe second FIFO in the second island. The data path extends is asubstantially straight line from the center part of the first island tothe center part of the second island. When a bus transaction valuepasses through the distributed credit FIFO and then through a crossbarswitch of the second island, an arbiter in the crossbar switch returns ataken signal. The taken signal passes back through the second chain ofregisters to the credit count circuit in the first island. The creditcount circuit maintains a credit count value for the distributed creditFIFO. The credit count circuit decreases a credit value when a datavalue is pushed into the first FIFO, and is increased when the takensignal reaches the credit count circuit. There is one such distributedcredit FIFO for the link portion from the first island to the secondisland, and there is another such distributed credit FIFO for the linkportion from the second island to the first island.

In a fourth novel aspect, an island-based network flow processor(IB-NFP) integrated circuit comprises six islands: 1) a first island (aMAC island) that converts incoming symbols into a packet; 2) a secondisland (first NBI island) that analyzes at least part of the packet andgenerates therefrom first information indicative of whether the packetis a first type of packet or a second type of packet, 3) a third island(ME island) that receives the first information and the header portionfrom the second island via a configurable mesh data bus, and thatgenerates second information indicating where the header portion isstored and where the payload portion is stored; a fourth island (MUisland) that receives a payload portion from the second island via theconfigurable mesh data bus; a fifth island (second NBI island) thatreceives second information from the third island via the configurablemesh data bus, that receives the header portion from the third islandvia the configurable mesh data bus, and that receives the payloadportion from the fourth island via the configurable mesh data bus, andthat performs egress scheduling; and a sixth island (second MAC island)that receives the header portion and the payload portion from the fifthisland and converts the header portion and the payload portion intooutgoing symbols. The first, second, third, fourth, fifth, and sixthislands all have the same rectangular size and shape. The first, second,third, fourth, fifth, and sixth islands all have the same configurablemesh data bus structure, configurable mesh control bus structure, andconfigurable mesh event bus structure.

In a fifth novel aspect, memories of an island-based network flowprocessor (IB-NFP) integrated circuit have a common interface structure.The island-based network flow processor integrated circuit has a firstisland and a second island. The first island comprises a first memoryand first data bus interface circuitry. The second island comprises aprocessor, a second memory, and second data bus interface circuitry. Thesecond memory is tightly coupled to the processor. The first data businterface circuitry is substantially identical to the second data businterface circuitry. The processor in the second island can issue acommand for a target memory to do an action. If a target field in thecommand has a first value then the target memory is the first memory inthe first island whereas if the target field in the command has a secondvalue then the target memory is the second memory in the second island.In one example, the command is a command on a configurable mesh databus. The command format is the same, regardless of whether the targetmemory is local or remote. If the target memory is remote, then a databus bridge in the first island adds destination information beforeputting the command onto the global configurable mesh data bus.

In a sixth novel aspect, a first packet is received onto a first islandof an island-based network flow processor (IB-NFP) integrated circuit.The header portion of the first packet is communicated to and is storedin a second island (for example, in a tightly coupled memory of aprocessor island). The payload portion of the first packet iscommunicated to and is stored in a third island (for example, a memoryisland). When the first packet is to be transmitted, the header portionis communicated from the second island to a fourth island, and thepayload portion is communicated from the third island to the fourthisland. The first packet is then output from the integrated circuit. Asecond packet is received onto the first island. The header portion ofthe second packet is communicated to and is stored in the second island.The payload portion of the second packet is communicated to and isstored in the third island. The header portion is communicated from thesecond island to the third island. When the second packet is to betransmitted, both the header portion and the payload portion of thesecond packet are communicated from the third island to the fourthisland, whereafter the second packet is output from the integratedcircuit. In one example, the header portion is not moved into the thirdisland unless memory resources in the second island are scarce, but ifmemory resources in the second island are scarce then the header portionis moved to be stored in the third island along with the payloadportion, thereby freeing up memory resources in the second island.

In a seventh novel aspect, an island-based network flow processor(IB-NFP) integrated circuit comprises a plurality of islands. Eachisland comprises a switch and four half links. The islands are coupledtogether such that the half links and the switches form a configurablemesh control bus. The configurable mesh control bus is configured tohave a tree structure such that configuration information passes fromthe switch of a root island to the switch of each of the other islands,and such that circuitry in each of the plurality of islands isconfigured by configuration information received via the configurablemesh control bus from the root island. In one example, the configurablecontrol mesh bus portion of each island includes a statically configuredswitch and multiple half links that radiate from the switch. The staticconfiguration is determined by hardwired tie off connections associatedwith the island. Configuration information communicated across the treestructure is used to configure a configurable mesh data bus of theisland-based network flow processor integrated circuit.

In an eighth novel aspect, an island-based network flow processor(IB-NFP) integrated circuit is configured to have a local event ring.The integrated circuit comprises a plurality of rectangular islandsdisposed in rows. Each rectangular island comprises a Run TimeConfigurable Switch (RTCS) and a plurality of half links. Therectangular islands are coupled together such that the half links andthe RTCSs together form a configurable mesh event bus. The configurablemesh event bus is configured to form the local event ring. The localevent ring provides a communication path along which an event packet iscommunicated to each rectangular island along the local event ring. Thelocal event ring involves event ring circuits and event ring segments.Upon each transition of a clock signal, an event packet moves throughthe ring from event ring segment to event ring segment. Eventinformation and not packet data travels through the ring. The localevent ring functions as a source-release ring in that only the eventring circuit that inserted the event packet onto the ring can delete theevent packet from the ring. An event ring circuit on the local eventring can only insert an event packet onto an event ring segment if thereis no event packet present on the event ring segment. A bit of the valueon an event ring segment indicates whether the remainder of the value onthe segment is an event packet or not.

In a ninth novel aspect, an island-based network flow processor (IB-NFP)integrated circuit is configured to have a global event chain. Theintegrated circuit comprises a plurality of rectangular islands disposedin rows. Each rectangular island comprises a Run Time ConfigurableSwitch (RTCS) and a plurality of half links. The rectangular islands arecoupled together such that the half links and the RTCSs together form aconfigurable mesh event bus. The configurable mesh event bus isconfigured to form a local event ring and the global event chain. Thelocal event ring provides a communication path along which an eventpacket is communicated to each rectangular island along the local eventring. The event packet can pass from the local event ring and onto theglobal event chain such that the event packet is then communicated alongthe global event chain. The global event chain is not a ring, but ratherextends in one example to one of islands and terminates in that island.In one example, the local event ring comprises a plurality of event ringcircuits and a plurality of event ring segments. One of the event ringsegments receives an event packet that is circulating in the local eventring, and determines if the event packet meets a criterion, and only ifthe packet meets the criterion does the event ring circuit pass theevent packet onto the global event chain. The criterion is determined byconfiguration information stored in the event ring circuit. Theconfiguration information can be changed via the configurable meshcontrol bus. A configurable event manager at the end of the global eventchain receives global event packets from the global event chain,analyzes the global event packets, and collects and logs informationabout the global event packets. How the event manager analyzes theglobal event packets is determined by configuration information storedin the event manager. The configuration information can be changed viathe configurable mesh control bus.

In a tenth novel aspect, an island-based network flow processor (IB-NFP)integrated circuit includes islands organized in rows. A configurablemesh event bus extends through the islands and is configured to form alocal event ring. The configurable mesh event bus is configured withconfiguration information received via a configurable mesh control bus.The local event ring involves event ring circuits and event ringsegments. In one example, a packet is received onto a first island. Ifan amount of a processing resource available to the first island isbelow a predetermined threshold, then an event packet is communicatedfrom the first island to a second island via the local event ring. Inresponse, the second island sends a first communication from the secondisland to a third island across a configurable mesh data bus. The thirdisland receives the first communication and in response sends a secondcommunication from the third island to the first island across theconfigurable mesh data bus. As a result of the first and secondcommunications, the amount of the processing resource available to thefirst island for handling incoming packet information is increased. Inone specific example, the processing resource is an amount of buffermemory that is available to the first island for storing incoming packetinformation. A buffer list identifies a plurality of memory buffers thatare available to the first island for storing incoming packetinformation. As a result of the first and second communications, anindication of an additional buffer or buffers is added to the bufferlist so that the first island has more buffer space available forstoring incoming packet information.

Further details and embodiments and techniques are described in thedetailed description below. This summary does not purport to define theinvention. The invention is defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, where like numerals indicate like components,illustrate embodiments of the invention.

FIG. 1 is a simplified diagram of an MPLS (MultiProtocol LabelSwitching) router 1.

FIG. 2 is a more detailed top-down diagram of IB-NFP integrated circuit12.

FIG. 3 shows a way of laying out and interconnecting circuitry in thedesign of an integrated circuit.

FIG. 4 is a diagram of one of the islands 82 of the pattern of FIG. 3.

FIG. 5 shows a way of laying out and interconnecting circuitry in thedesign of an integrated circuit in accordance with one novel aspect.

FIG. 6 is a diagram of one of the islands 99 of the pattern of FIG. 5.

FIG. 7 is a diagram that illustrates the circuitry for the configurablemesh data bus that is part of a full island.

FIG. 8 is a diagram that illustrates the circuitry of the configurablemesh control bus that is part of a full island.

FIG. 9 is a diagram that illustrates the circuitry of the configurablemesh event bus that is part of a full island.

FIG. 10 is a more detailed diagram of the structure of one of the meshbuses of the configurable mesh data bus in a part of IB-NFP integratedcircuit 12.

FIG. 11 is a diagram that shows the crossbar switch CB3 of island I3(61).

FIG. 12 is a diagram of island I3 that illustrates an operation of LUT118 of crossbar switch CB3 in further detail.

FIG. 13 is a diagram that shows the crossbar switch CB3 in furtherdetail.

FIG. 14 is a diagram that illustrates the terminology employed.

FIG. 15 is a diagram that shows how link portion 136 between islands I1and I2 is realized as a distributed credit FIFO.

FIG. 16 is a top-down diagram that illustrates in schematic fashion howisland I3 includes a separate set of half link portions and crossbarswitches for the four different mesh buses of the configurable mesh CPPdata bus.

FIG. 17 is a flowchart of a write operation method 1000 that might occuracross the configurable mesh CPP data bus.

FIG. 18 is a diagram of the formal of a bus transaction value thatpasses over the configurable mesh data bus.

FIG. 19 is a table describing the payload of a bus transaction value inthe situation in which the bus transaction value is a command.

FIG. 20 is a table describing the payload of a bus transaction value inthe situation in which the bus transaction value is a pull-id.

FIG. 21 is a table describing the payload of a bus transaction value inthe situation in which the bus transaction value is data pull or push.

FIG. 22 is a table describing the payload of a bus transaction value inthe situation in which the bus transaction value is data pull.

FIG. 23 is a table describing the payload of a bus transaction value inthe situation in which the bus transaction value is data push.

FIG. 24 is a flowchart of a read operation method 2000 that might occuracross the configurable mesh CPP data bus.

FIG. 25 is a diagram the configurable mesh control bus structure of theIB-NFP integrated circuit 12.

FIG. 26 is a top-down diagram that illustrates the configurable meshdata bus and how the islands do not abut each other in one embodiment.

FIG. 27 is a top-down diagram that illustrates the configurable meshcontrol bus and how the islands do not abut each other in oneembodiment.

FIG. 28 is a top-down diagram that illustrates the configurable meshevent bus and how the islands do not abut each other in one embodiment.

FIG. 29 is a top-down diagram of the configurable mesh control busstructure of the IB-NFP integrated circuit 12 configured to form a treestructure.

FIG. 30 is a simplified top-down diagram of the control bus portion ofthe ARM island 51 and parts of ME island 53 and ME island 54.

FIG. 31 is a diagram that shows how configuration information flows intoand through ME island 53.

FIG. 32 is a simplified diagram of the central switch of the control busstructure of an island.

FIG. 33 is a diagram of the configurable mesh event bus structure of theIB-NFP integrated circuit 12.

FIG. 34 is a simplified perspective view of the local event ring withinNBI island 72.

FIG. 35 is a functional diagram of the RTCS 206 shown in FIG. 34.

FIG. 36 is a more detailed block diagram of event ring circuit 209 shownin FIG. 34.

FIG. 37 is a more detailed block diagram of event ring circuit 210 shownin FIG. 34.

FIG. 38A is a diagram of an event packet bit sequence.

FIG. 38B is a signal table that identifies and explains the variousparts of an event packet.

FIG. 39 is a diagram of the configurable mesh event bus configured toform two local event rings and a global event chain.

FIG. 40 is a simplified system level illustration that shows how a localevent ring is a source-release ring.

FIG. 41 is a simplified diagram of the first 245 and second 246 localevent rings and single global event chain 247 shown in FIG. 39.

FIG. 42 is a more detailed block diagram of the event manager 251 withinisland 51.

FIG. 43 is a schematic diagram that illustrates an operational exampleof IB-NFP integrated circuit 12 within the MPLS router 1 of FIG. 1.

FIG. 44 is a diagram of the four SerDes I/O blocks 19-22 and the ingressMAC island 71 of IB-NFP integrated circuit 12.

FIG. 45 is a diagram that illustrates how a packet is communicated as asequence of minipackets across connections 312.

FIG. 46 is a diagram of ingress NBI island 72.

FIG. 47 is a table that sets forth the parts of preclassificationresults 321.

FIG. 48 is a table that sets forth the parts of an ingress packetdescriptor.

FIG. 49 is a table that sets forth the parts of an egress packetdescriptor.

FIG. 50 is a diagram of the microengine (ME) island 66.

FIG. 51 is a bit sequence map of a bus transaction value used tocommunicate packet data from the ingress NBI island 72 to the ME island66 across the CPP data bus.

FIG. 52 is a diagram of MU half island 68 and associated SRAM block 78.

FIG. 53 is a diagram of egress NBI island 63.

FIG. 54 is a diagram of egress MAC island 64 and SerDes blocks 25-28.

FIG. 55 is a diagram that illustrates a packet flow in the operationalexample when local memory resources in the CTM 333 of the ME island 66are determined not to be scarce (for example, the processing resource isdetermined not to be below a predetermined threshold).

FIG. 56 is a diagram that illustrates a packet flow in the operationalexample when local memory resources in the CTM 333 of the ME island 66are determined to be scarce (for example, the processing resource isdetermined to be below a predetermined threshold).

FIG. 57 is a diagram that illustrates the use of a local event ring anda configurable mesh data bus for flow control in the IB-NFP integratedcircuit 12.

DETAILED DESCRIPTION

Reference will now be made in detail to background examples and someembodiments of the invention, examples of which are illustrated in theaccompanying drawings. In the description and claims below, relationalterms such as “horizontal”, “vertical”, “lateral”, “top”, “upper”,“bottom”, “lower”, “right”, “left”, “over” and “under” may be used todescribe relative orientations between different parts of a structurebeing described, and it is to be understood that the overall structurebeing described can actually be oriented in any way in three-dimensionalspace.

FIG. 1 is a simplified diagram of an MPLS (MultiProtocol LabelSwitching) router 1. Router 1 includes a backplane 2, a management card3, and line cards 4-6. Each of the line cards can receive 100 Gbps(gigabits per second) packet traffic from another network via a fiberoptic cable 7 and also can transmit 100 Gbps packet traffic to anothernetwork via another fiber optic cable 8. In addition, each line card canreceive 100 Gbps packet traffic from the switch fabric 9 of thebackplane and can also transmit 100 Gbps packet traffic to the switchfabric. Line cards 4-6 are of identical construction. In this example,flows of packets are received into line card 4 from a network via thefiber optic cable 7 or from the switch fabric 9. Certain functions thenneed to be performed on the line card including looking up MPLS labels,determining destinations for incoming flows of packets, and schedulingthe transmitting of flows of packets. Packets of the flows pass from theline card 4 and out either to the network via optical cable 8 or to theswitch fabric 9.

Line card 4 includes a first optical transceiver 10, a first PHYintegrated circuit 11, an Island-Based Network Flow Processor (IB-NFP)integrated circuit 12, a configuration Programmable Read Only Memory(PROM) 13, an external memory such as Dynamic Random Access Memory(DRAM) 40-41, a second PHY integrated circuit 15, and a second opticaltransceiver 16. Packet data received from the network via optical cable7 is converted into electrical signals by optical transceiver 10. PHYintegrated circuit 11 receives the packet data in electrical form fromoptical transceiver 10 via connections 17 and forwards the packet datato the IB-NFP integrated circuit 12 via SerDes connections 18. In oneexample, the flows of packets into the IB-NFP integrated circuit fromoptical cable 7 is 100 Gbps traffic. A set of four SerDes circuits 19-22within the IB-NFP integrated circuit 12 receives the packet data inserialized form from SerDes connections 18, deserializes the packetdata, and outputs packet data in deserialized form to digital circuitrywithin IB-NFP integrated circuit 12.

Similarly, IB-NFP integrated circuit 12 may output 100 Gbps packettraffic to optical cable 8. The set of four SerDes circuits 19-22 withinthe IB-NFP integrated circuit 12 receives the packet data indeserialized form from digital circuitry within integrated circuit 12.The four SerDes circuits 19-22 output the packet data in serialized formonto SerDes connections 23. PHY 15 receives the serialized form packetdata from SerDes connections 23 and supplies the packet data viaconnections 24 to optical transceiver 16. Optical transceiver 16converts the packet data into optical form and drives the opticalsignals through optical cable 8. Accordingly, the same set of fourduplex SerDes circuits 19-22 within the IB-NFP integrated circuit 12communicates packet data both into and out of the IB-NFP integratedcircuit 12.

IB-NFP integrated circuit 12 can also output packet data to switchfabric 9. Another set of four duplex SerDes circuits 25-28 within IB-NFPintegrated circuit 12 receives the packet data in deserialized form, andserializes the packet data, and supplies the packet data in serializedform to switch fabric 9 via SerDes connections 29. Packet data fromswitch fabric 9 in serialized form can pass from the switch fabric viaSerDes connections 30 into the IB-NFP integrated circuit 12 and to theset of four SerDes circuits 25-28. SerDes circuits 25-28 convert thepacket data from serialized form into deserialized form for subsequentprocessing by digital circuitry within the IB-NFP integrated circuit 12.

Management card 3 includes a CPU (Central Processing Unit) 31. CPU 31handles router management functions including the configuring of theIB-NFP integrated circuits on the various line cards 4-6. CPU 31communicates with the IB-NFP integrated circuits via dedicated PCIEconnections. CPU 31 includes a PCIE SerDes circuit 32. IB-NFP integratedcircuit 12 also includes a PCIE SerDes 33. The configuration informationpasses from CPU 31 to IB-NFP integrated circuit 12 via SerDes circuit32, SerDes connections 34 on the backplane, and the PCIE SerDes circuit33 within the IB-NFP integrated circuit 12.

External configuration PROM (Programmable Read Only Memory) integratedcircuit 13 stores other types of configuration information such asinformation that configures various lookup tables on the IB-NFPintegrated circuit. This configuration information 35 is loaded into theIB-NFP integrated circuit 12 upon power up. As is explained in furtherdetail below, IB-NFP integrated circuit 12 can store various types ofinformation including buffered packet data in external DRAM integratedcircuits 40-41.

FIG. 2 is a more detailed top-down diagram of IB-NFP integrated circuit12. IB-NFP integrated circuit 12 includes many I/O (input/output)terminals (not shown). Each of these terminals couples to an associatedterminal of the integrated circuit package (not shown) that houses theIB-NFP integrated circuit. The integrated circuit terminals may beflip-chip microbumps and are not illustrated. Alternatively, theintegrated circuit terminals may be wire bond pads.

SerDes circuits 19-22 are the first set of four SerDes circuits that areused to communicate with the external network via the optical cables 7and 8. SerDes circuits 25-28 are the second set of four SerDes circuitsthat are used to communicate with the switch fabric 9. Each of theseSerDes circuits is duplex in that it has a SerDes connection forreceiving information and it also has a SerDes connection fortransmitting information. Each of these SerDes circuits can communicatepacket data in both directions simultaneously at a sustained rate of 25Gbps. IB-NFP integrated circuit 12 accesses external memory integratedcircuits 36-41 via corresponding 32-bit DDR physical interfaces 42-47,respectively. IB-NFP integrated circuit 12 also has several generalpurpose input/output (GPIO) interfaces. One of these GPIO interfaces 48is used to access external PROM 13.

In addition to the area of the input/output circuits outlined above, theIB-NFP integrated circuit 12 also includes two additional areas. Thefirst additional area is a tiling area 49 of islands 50-74. Each of theislands is either of a full rectangular shape, or is half the size ofthe full rectangular shape. For example, the island 55 labeled “PCIE(1)” is a full island. The island 60 below it labeled “ME CLUSTER (5)”is a half island. The functional circuits in the various islands of thistiling area 49 are interconnected by: 1) a configurable mesh CPP databus, 2) a configurable mesh control bus, and 3) a configurable meshevent bus. Each such mesh bus extends over the two-dimensional space ofislands with a regular grid or “mesh” pattern. These mesh buses aredescribed in further detail below.

In addition to tiling area 49, there is a second additional area oflarger sized blocks 75-79. The functional circuitry of each of theseblocks is not laid out to consist of islands and half-islands in the waythat the circuitry of tiling area 49 is laid out. The mesh busstructures do not extend into or over any of these larger blocks. Themesh bus structures do not extend outside of tiling area 49. Thefunctional circuitry of a larger sized block outside the tiling area 49may connect by direct dedicated connections to an interface islandwithin tiling area 49 and through the interface island achieveconnectivity to the mesh buses and other islands.

FIG. 3 shows a way of laying out and interconnecting circuitry in thedesign of an integrated circuit. The functional circuitry is partitionedinto blocks 80-95 referred to as tiles or islands. To provideinterconnectivity between any selected two of the islands, aconfigurable mesh bus of horizontal links, vertical links, links down tofunctional circuitry in each island, and crossbar switches is provided.FIG. 4 is a diagram of one of the islands 82. Its crossbar switch 96 iscentrally located. A first link is provided from the functional circuitat port location P5 to the crossbar switch 96. Information can becommunicated from the functional circuitry of the island to the crossbarswitch via this link. A second link is provided from the crossbar switchto the functional circuitry at port location P6. Information can becommunicated from the crossbar switch to the functional circuitry viathis second link. A half link extends vertically from the crossbarswitch to a first port location P1 in the center of the top edge of theisland. A second half link extends horizontally from the crossbar switchto a second port location P2 at the right edge of the island. A thirdhalf link extends vertically from the crossbar switch to a third portlocation P3 in the center of the bottom edge of the island. A fourthhalf link extends horizontally from the crossbar switch to a fourth portlocation P4 at the right edge of the island. Information coming into theisland on one of the half links can be switched by the crossbar switchso that the information then passes out of the island on a selected oneof the other half links of the island or so that the information issupplied to the functional circuitry of the island. As illustrated inFIG. 3, the islands are laid out in regular rows and columns so that thehalf links align with one another and form links between crossbarswitches. This tiling pattern may be referred to as a Manhattan pattern.If a connection is to be provided from functional circuitry in the lowerleft island 93 to functional circuitry in island 82 in the top row, thenthe connection would pass through a minimum of four intervening islandsbefore reaching island 82. The path of such a connection is indicated bydashed line 97.

FIG. 5 shows a way of laying out and interconnecting circuitry in thedesign of an integrated circuit in accordance with one novel aspect. Thefunctional circuitry is partitioned into rectangular islands 98-113 asillustrated. FIG. 6 is a diagram of one of the islands 99. A crossbarswitch 114 is centrally located. Four half links are not, however,oriented to form a cross-shape as in FIG. 4, but rather six half linksare oriented to form a star-shape as illustrated as in FIG. 6. A firstlink is provided from the functional circuitry at port location P7 tothe crossbar switch 114. A signal can be supplied from the functionalcircuitry to the crossbar switch via this link. A second link isprovided from the crossbar switch 114 to the functional circuitry atport location P8. A signal can be supplied from the crossbar switch tothe functional circuitry via this second link. A first half link extendsdiagonally from the crossbar switch 114 up and to the left to a firstport location P1. A second half link extends diagonally from thecrossbar switch 114 up and to the right to a second portion location P2.A third half link extends horizontally from the crossbar switch 114 tothe right to a third port location P3. A fourth half link extendsdiagonally from the crossbar switch 114 down and to the right to afourth port location P4. A fifth half link extends diagonally fro thecrossbar switch 114 down and to the left to a fifth port location P5. Asixth half link extends horizontally from the crossbar switch 114 to theleft to a sixth port location P6. The rectangular islands of FIG. 5 arelaid out in rows, but the side edges of the islands of one row arestaggered laterally with respect to the corresponding side edges of theislands of the next row. The islands are therefore oriented in astaggered brick structure as illustrated. If a connection is to beprovided from functional circuitry in the lower left island 110 tofunctional circuitry in the island 100 in the top row, then theconfigurable mesh bus structure can be configured such that a connectionis established between these two islands that only passes through threeintervening islands before reaching island 100. The path of such aconnection is indicated by dashed line 115. Connectivity is improved ascompared to the Manhattan pattern of FIG. 3. The comparative diagrams ofFIGS. 3 and 5 are only for a small number of islands, but thecomparative advantage of the staggered pattern of FIG. 5 as compared tothe Manhattan pattern of FIG. 3 becomes more pronounced as the layoutpatterning technique is extended to include more and more islands.

The islands of the tiling area 49 of the IB-NFP integrated circuit 12are disposed in the staggered pattern explained in connection with FIG.5. Each island includes: 1) an amount of functional circuitry, 2)circuitry for a configurable mesh data bus, 3) circuitry for aconfigurable mesh control bus, and 4) circuitry for a configurable meshevent bus.

FIG. 7 is a diagram that illustrates the circuitry for the configurablemesh data bus that is part of a full island. The configurable mesh databus structure actually includes four mesh bus structures, each of whichincludes a crossbar switch that is disposed in the center of the islandas illustrated, and each of which includes six half links that extend toport locations P1-P6 as illustrated, and each of which also includes twolinks that extend between the crossbar switch and the functionalcircuitry of the island at port locations P7 and P8. These four mesh busstructures are referred to as the command mesh bus, the pull-id meshbus, and data0 mesh bus, and the data1 mesh bus. The half links of thesemesh buses do not necessarily all extend along the exact same path whenthe island is considered from the top-down perspective, but rather thehalf links of these mesh buses extend roughly along the same line. Themesh buses terminate at the edges of the island such that if anotheridentical tile were laid out to be adjacent, then the half links of thecorresponding mesh buses of the two islands would align and couple toone another in an end-to-end collinear fashion to form the staggeredpattern illustrated in FIG. 5. Similarly, the circuitry of the crossbarswitches of the four mesh buses are not all disposed in the same exactlocation when the island is considered from the top-down perspective,but rather the crossbar switches are located roughly in the center ofthe island. The four mesh buses of the configurable mesh data bustogether are a Command/Push/Pull (CPP) bus as is explained in furtherdetail below.

FIG. 8 is a diagram that illustrates the circuitry of the configurablemesh control bus that is part of a full island. Unlike the structure ofthe mesh buses of the configurable mesh data bus of FIG. 7 where eachcrossbar switch is coupled to the functional circuitry of the island bytwo links, the crossbar switch of the configurable mesh control bus ofFIG. 8 is coupled to the functional circuitry of the island by one link.Also, the configurable mesh control bus circuitry does not involvehorizontally extending half links. The crossbar switch is configured byhardwired tie off connections as described in further detail below. Thisconfiguration is fixed at the time of chip manufacture and is notchangeable. Accordingly, the crossbar switch is sometimes referred tohere more specifically as a “statically configured switch”, rather thanas a crossbar switch.

FIG. 9 is a diagram that illustrates the circuitry of the configurablemesh event bus that is part of a full island. The structure of the eventbus is similar to the structure of the control bus, except that thecrossbar switches of the configurable mesh event bus are not hardwiredinto one configuration. The crossbar switch is referred to as a “runtime configured switch”.

FIG. 10 is a more detailed diagram of the structure of one of the meshbuses of the configurable mesh data bus in a part of IB-NFP integratedcircuit 12. A first crossbar CB1 is located centrally within a firstrectangular island I1. The first rectangular island I1 is PCIE (1)island 55 of FIG. 2. A second crossbar CB2 that is located centrallywithin a second rectangular island I2. The second rectangular island I2is ME cluster (3) island 56 of FIG. 2. A third crossbar CB3 that islocated centrally within a third rectangular island I3. The thirdrectangular island I3 is Crypto Bulk (2) island 61 of FIG. 2. The firstand the second islands I1 and I2 are disposed in a first row thatextends along a horizontal dimension. The third island I3 is disposed ina second row that extends from the first row along a vertical dimension(i.e., the second row is below the first row when considered from theperspective of FIG. 10). A first link L1 comprises a first half linkHL11 and a second half link HL12. HL11 and HL12 are collinear. Firstlink L1 extends in a substantially straight line between CB1 and CB2. Asecond link L2 comprises a third half link HL21 and a fourth half linkHL22. Half links HL21 and HL22 are collinear. Second link L2 extends ina substantially straight line between CB1 and CB3. A third link L3comprises a fifth half link HL31 and a sixth half link HL32. Half linksHL31 and HL32 are collinear. Third link L3 extends in a substantiallystraight line between CB2 and CB3. Links L1, L2 and L3 form an isoscelestriangle as illustrated. Reference numerals P1, P2, P3, P4, P5 and P6identify port locations. Island I1 has an upper left corner C1, an upperright corner C2, a lower right corner C3, and a lower left corner C4.Port location P1 is located on the top edge of the island about onequarter of the way from the upper left corner to the upper right corner.Port location P2 is located on the top edge of the island about threequarters of the way from the upper left corner to the upper rightcorner. Port location P3 is located in the middle of the right edge ofthe island. Port location P4 is located on the bottom edge of the islandabout one quarter of the way from the lower right corner to the lowerleft corner. Port location P5 is located on the bottom edge of theisland about three quarters of the way from the lower right corner tothe lower left corner. Port location P6 is located in the middle of theleft edge of the island.

Island I6 is a half island. Island I6 includes a centrally locatedcrossbar switch CB6. Two links extend between crossbar switch CB6 andthe functional circuitry of the half island. There are, however, onlythree half links that extend to three corresponding port locations P1,P2 and P3. Half link HL42, for example, is oriented to extend incollinear fashion to half link HL41 of island I1. Half link HL42 joinsat its port location P1 with half link HL41 at its port location P5. Theother connections between the half links of the various islands are notdescribed here. There are four instances of the general bus structureillustrated in FIG. 10, one for the command mesh of the configurablemesh data bus, one for the pull-id mesh of the configurable mesh databus, one for the data0 mesh of the configurable mesh data bus, and onefor the data1 mesh of the configurable mesh data bus.

FIG. 11 is a diagram that shows the crossbar switch CB3 of island I3(61). Each half link is bidirectional in that it can supply informationfrom an input port location to the crossbar switch CB3, and in that itcan receive information from the crossbar switch CB3 and supply thatinformation to an output port location. For example, half link HL51includes a half link portion 116 that can communicate information frominput port location P6 to CB3, and half link HL51 also includes a halflink portion 117 that can communicate information from CB3 to outputport location P6. There is a Look Up Table (LUT) within CB3 associatedwith each half link portion that supplies information to the crossbarswitch. In the case of FIG. 11, LUT 118 is the LUT associated withincoming half link portion 116. There is also an arbiter within CB3associated with each half link portion that communicates informationfrom the crossbar switch to an output port location. In the case of FIG.11, arbiter 183 is the arbiter associated with half link portion 117that communicates information from CB3 to output port location 6.

FIG. 12 is a diagram of island I3 that illustrates the operation of LUT118 in further detail. Crossbar switch CB3 receives information fromhalf link portion 116 and switches the information so that it is outputto another half link portion. In the case of FIG. 12, the information isswitched so that it is output to the half link portion for output portlocation 4. Incoming half link portion 116 includes a chain of registers119 and an input FIFO (First In First Out) 120. The multiple bits ofinformation being communicated across the half link portion 116 iscalled a bus transaction value. The bus transaction value includes afinal destination value portion, a valid bit, and a payload portion. Thefinal destination value indicates the destination island to which thebus transaction value will be communicated through configurable mesh CPPdata bus. Input FIFO 120 supplies a bus transaction value to CB3. Thefinal destination value portion of the bus transaction value is suppliedto LUT 118. LUT 118 was previously configured so that when it ispresented with a particular final destination value 121, it outputs aFIFO lookup value 122 that causes CB3 to route the bus transaction valuethrough CB3 to an appropriate output port. In the example of FIG. 12,the final destination value is 13. From the value 13 the LUT 118 outputsa lookup value of 4. By configuring all the LUTs appropriately, a bustransaction value having a particular final destination value will berouted through the configurable mesh data bus via one and only one pathto the island indicated by the final destination value. In the exampleof FIG. 12, LUT 118 outputs a lookup value of 4. CB3 uses this lookupvalue 4 to route the bus transaction value onto the output link portionthat will communicate the bus transaction value to output port 4.

FIG. 13 is a diagram that shows the crossbar switch CB3 in furtherdetail. In the left column are illustrated the input FIFOs. There is onesuch input FIFO at the end of each of the eight half link portions thatcan supply bus transaction values from an input port to CB3. Forexample, the half link portion that extends from input port 6 ends ininput FIFO 120. LUT 118 is associated with this incoming half linkportion. The final destination value portion of the bus transactionvalue output by FIFO 120 is supplied to LUT 118. The bus transactionvalue output by FIFO 120 is also communicated horizontally into thecrossbar switch via a set of conductors 123. Similarly, the half linkportion that extends from input port 3 ends in FIFO 124. The finaldestination value of the bus transaction value output by FIFO 124 issupplied to LUT 125. The bus transaction value output by FIFO 124 isalso communicated horizontally into the crossbar switch via a set ofconductors 126. As illustrated, there is one set of horizontallyextending conductors for each of the eight input ports.

In addition to the sets of horizontally extending conductors, there isalso one set of vertically extending conductors for each of the eighthalf link portions onto which a bus transaction value can be routed. Forexample, the set of vertical conductors 127 can communicate a bustransaction value vertically downward to the output FIFO 128 of the halflink portion that extends to output port 4. There is a set of switchesat the intersection of each set of horizontally extending conductors andeach set of vertically extending conductors. Each such set of switchesis generally just referred to as a switch. Each such switch can beturned on by an arbiter to couple the set of horizontally extendingconductors to the set of vertically extending conductors. Alternatively,each such switch can be turned off by an arbiter so that the switch doesnot couple the set of horizontally extending conductors to the set ofvertically extending conductors. There is an arbiter for each of theeight output ports as illustrated. The arbiter associated with an outputport controls the eight switches along the set of vertically extendingconductors that extend to the output FIFO for that output port. In theexample illustrated, arbiter 131 can turn on switch 129 to couple theset of horizontally extending conductors 123 to the set of verticallyextending conductors 127. Also, arbiter 131 can turn on switch 130 tocouple the set of horizontally extending conductors 126 to the set ofvertically extending conductors 127. There are eight such switches, onefor each input port, and each of these switches is controlled by arbiter131.

A 1.0 GHz global clock clocks the data bus structure, including thearbiters. During a given period of the 1.0 GHz global clock, at most onebus transaction value is switched onto a set of vertically extendingconductor for output to a particular output port. A contention situationmay arise in which, during a given period of the global clock, the bustransaction values of multiple sets of horizontally extending conductorsare to be switched onto the same output port. For example, two bustransaction values on the sets of horizontally extending conductors 123and 126 may have final destination values. When looked up in theirassociated LUTs, both final destinations result in the both bustransactions values being destined to go out to output port 4. If,however, both switches 129 and 130 were made to be conductive during thesame global clock period, then a collision would occur and the data ofthe bus transaction values would be corrupted. The arbiter 131 preventsthis problem. If there is space available in the output FIFO 128associated with arbiter 131, then arbiter 131 causes one bus transactionvalue to be pushed into output FIFO 128. The arbiter receives all LUTlookup values from the LUTs of the input ports. If the lookup valuesindicate that only one bus transaction value is to be routed onto theoutput half link portion for output port 4, then the arbiter 131controls the appropriate switch to turn on and to couple the bustransaction value onto the vertically extending conductors 127. If,however, the lookup values from the LUTs indicate that two or more bustransaction values are to be routed to output port 4, then the arbiter131 turns on the switches so that a bus transaction value from only oneof the input ports will be switched onto the set of vertically extendingconductors and to the output port. In the example of FIG. 13, the finaldestination values of the two bus transaction values received from inputports 3 and 6 both result in their respective LUTs outputting lookupvalues indicating that their respective bus transaction values are to berouted to output port 4. Arbiter 131 receives the lookup values from alleight LUTs. In the notation of FIG. 13, the OP=4 notation means “outputport 4”. Each LUT actually supplies a single bit to each of the eightarbiters. If the bit to an arbiter is asserted, this indicates to thearbiter that the LUT for the corresponding input port is signaling thata bus transaction value should be routed to the output port associatedwith the arbiter. In the example of FIG. 13, LUTs 118 and 125 bothassert their respective single bit OP values for arbiter 131. Inresponse, arbiter 131 controls switch 129 to be on during a first globalclock period while switch 130 is off. After the bus transaction valuefrom input port location 6 has passed through the crossbar switch andout to output port location 4 during the first global clock period, thenthe arbiter 131 switches switch 129 to be off and switch 130 to be onduring the second global clock period. The bus transaction value frominput port location 3 then passes through the crossbar switch and tooutput port location 4 during the second global clock cycle.

FIG. 14 is a diagram that illustrates the terminology employed. Link L1is the bidirectional link that extends horizontally between crossbarswitch CB 1 of island I1 and crossbar switch CB2 of island I2 of FIG.10. This link L1 extends through port location 3 of island I1, andthrough port location 6 of island I2. The bottom portion of FIG. 14shows that this link involves two half links. Half link HL11 is the halfof the link in island I1. Half link HL12 is the half of the link inisland I2. Each half link is bidirectional. Each half link includes twohalf link portions. For example, half link portion 132 extends from CB1to port location 3 of island I1. Half link portion 133 extends from portlocation 3 of island I1 to CB1. Half link portion 134 extends from portlocation 6 of island I2 to CB2. Half link portion 135 extends from CB2to port location 6 of island I2. Half link portions 132 and 134 arecollinear and join to form link portion 136. Half link portions 133 and135 are collinear and join to form link portion 137. The half linkportions that carry data away from a crossbar switch start with anoutput FIFO. FIFO 138 is the output FIFO for half link portion 132. FIFO140 is the output FIFO for half link portion 135. The half link portionsthat carry data into a crossbar switch end with an input FIFO. FIFO 139is the input FIFO for half link portion 133. FIFO 159 is the input FIFOfor half link portion 134.

FIG. 15 is a diagram that shows how link portion 136 between islands I1and I2 is realized as a distributed credit FIFO. Link portion 136includes the first FIFO 138 that is associated with output port 3 offirst island I1, a first chain of registers 142-145, second FIFO 159that is associated with input port 6 of second island I2, a second chainof registers 146-150, and a credit count circuit 151 disposed in thefirst island I1. As explained above, a bus transaction value can becommunicated from the crossbar switch CB1 of island I1, through outputFIFO 138, through chain of registers 142-145, and to input FIFO 159, andto crossbar switch CB2 of island I2. The chain of registers is clockedby the 1.0 GHz global clock signal 152. This global clock clocks all theregisters of all the links of the entire configurable mesh data bus. Thephysical distance between the crossbar switches CB1 and CB2 is smallerthan the distance a signal can propagate on a conductor across theintegrated circuit in one period of the global clock. Multiple bustransaction values may flow in a pipelined fashion through this linkportion.

Arbiter 154 is the arbiter of CB2 for output port 4. The other arbitersfor the other output ports are not illustrated. Arbiter 154 receives aFIFO full signal 141 from the output FIFO 153 for output port 4. If theFIFO full signal 141 indicates that space is available in output FIFO153, then arbiter 154 can cause a selected bus transaction value fromone of the input ports of the crossbar switch to be supplied through thecrossbar switch and into output FIFO 153. For each input FIFO of thecrossbar switch, its associated LUT receives the final destination valueof the bus transaction value being output by the associated input FIFO.A single conductor extends from each LUT to arbiter 154 for output port4. If the lookup value output by the LUT indicates that the bustransaction value should be output onto output port 4, then the LUTasserts the single-bit signal on this conductor for output port 4.Arbiter 154 receives one such single-bit signal from each LUT. All theconductors for communicating these single-bit signals are not shown inFIG. 15 due to space limitations. Signal 177 is the single-bit signalsupplied by LUT 178 to arbiter 154. If one of these OP signals indicatesthat a bus transaction value is to be output from output port 4, thenarbiter 154 turns on the appropriate switch to couple the bustransaction value from its horizontally extending conductors onto thevertically extending conductors and to output FIFO 153. A signalconductor extends from arbiter 154 to each of the eight switches thatmight be turned on to route a bus transaction value to output FIFO 153.Arbiter 154 can turn on a selected one of these eight switches byasserting the ON signal on the appropriate one of these eight signalconductors. In the case of FIG. 15, conductor 179 is the conductor thatextends from arbiter 154 to the switch that can switch input port 6 tooutput port 4. Reference numeral 180 identifies the set of horizontallyextending conductors for input port 6. Reference numeral 181 identifiesthe set of vertically extending conductors for output port 4. Referencenumeral 182 identifies the switch that can couple the set ofhorizontally extending conductors 180 to the set of vertically extendingconductors 181.

When the arbiter 154 causes a bus transaction value from input port 6 tobe routed through the crossbar switch and to be pushed into output FIFO153, the arbiter 154 issues a pop signal 163 back to input FIFO 159 fromwhich the bus transaction value came. In addition, the pop signal issupplied to the first register 146 of the second chain of registers. Thesecond chain of registers is really a chain of flip-flops that forms ashift register. This shift register shifts the pop signal (also called a“taken signal”) back to the left from the second island I2 to the creditcount circuit 151. When the taken signal 163 arrives at the credit countcircuit 151, a credit count value maintained by the credit count circuitis incremented to indicate that link portion 136 can hold one more bustransaction value. When the crossbar switch CB1 of the first island I1pushes a bus transaction value into FIFO 138 of link portion 136, thepush FIFO signal 164 is also supplied to credit count circuit 151. Thispush signal 164 causes credit count circuit 151 to decrement the creditvalue. A lower credit value indicates that link portion 136 can hold oneless bus transaction value. In the illustrated example, only if thecredit count is greater than zero can a data value be pushed into inputFIFO 138.

It is possible that the input FIFOs leading to the crossbar switchpresent multiple bus transaction values that should be switched to thesame output FIFO 153. If such a case of contention, arbiter 154 selectsone of the bus transaction values to be switched to the output FIFOduring the global clock period as explained above. The arbiter onlyturns on the appropriate switch to direct this one bus transaction valueto the output FIFO 153. The arbiter pops the input FIFO that suppliedthe bus transaction value at the end of the global clock period. Then inthe next global clock period, if there is space available in the outputFIFO as indicated by the FIFO full signal, then the arbiter turns on theappropriate switch to direct the next bus transaction value into theoutput FIFO. Once again, at the end of the transfer, the arbiter 154asserts the POP signal to the particular input FIFO that supplied thebus transaction value that was transferred. There are eight conductorsextending from arbiter 154 to the input FIFOs so that arbiter 154 cansupply a POP signal to the appropriate input FIFO. Only three of theseconductors are shown in FIG. 15 due to space limitations. In this way,the arbiter 154 only supplies one bus transaction value from one inputFIFO onto the output FIFO 153 during a given global clock period.

FIG. 16 is a top-down diagram that illustrates in schematic fashion howisland I3 includes a separate set of link portions and crossbar switchesfor the four different mesh buses of the configurable mesh CPP data bus.There is one set of half links and a crossbar switch for the commandmesh of the configurable mesh data bus. There is one set of half linksand a crossbar switch for the pull-id mesh of the configurable mesh databus. There is one set of half links and a crossbar switch for the data0mesh of the configurable mesh data bus. There is one set of half linksand a crossbar switch for the data1 mesh of the configurable mesh databus. The termination points of these half links at the edges of theisland are such that when two identical islands are made to be adjacent,that the command mesh half links join to another, that the pull-id meshhalf links join to one another, that the data0 mesh half links join toone another, and that the data1 mesh half links join to one another. Thefour mesh structures of the configurable mesh data bus implement acommand/push/pull (CPP) data bus.

FIG. 17 is a flowchart of a write operation method 1000 that might occuracross the configurable mesh CPP data bus. In a first step (step 1001),certain functional circuitry in one of the islands uses its data businterface to output a bus transaction value onto the configurable meshCPP data bus. This functional circuitry is referred to as the “master”of the write operation. The format of the bus transaction value is asset forth in FIG. 18. A bus transaction value 1006 includes a metadataportion 1007 and a payload portion 1008 as shown. The metadata portion1007 includes a final destination value 1009 and a valid bit 1010.

The bus transaction value in this case is a write command to write datainto functional circuitry in another island. The functional circuitrythat receives the bus transaction value and the data to be written isreferred to as the “target” of the write operation. The write command issaid to be “posted” by the master circuit onto the command mesh. Asindicated in FIG. 18, the write command includes a metadata portion anda payload portion. The metadata portion includes the 6-bit finaldestination value. This final destination value identifies an island bynumber, where the island identified is the final destination of the bustransaction value. The final destination value is used by the variouscrossbar switches of the command mesh to route the bus transaction value(i.e., the command) from the master circuit to the appropriate targetcircuit. All bus transaction values on the data bus that originate fromthe same island that have the same final destination value will traversethrough the configurable mesh data bus along the same one path all theway to the indicated final destination island.

A final destination island may have more than one potential targetcircuit. The 4-bit target field of payload portion indicates which oneof these targets in the destination island it is that is the target ofthe command. The 5-bit action field of the payload portion indicatesthat the command is a write. The 14-bit data reference field is areference usable by the master circuit to determine where in the masterthe data is to be found. The address field indicates an address in thetarget where the data is to be written. The length field indicates theamount of data.

In a next step (step 1002) in the method of FIG. 17, the target circuitreceives the write command from the command mesh and examines thepayload portion of the write command. From the action field the targetcircuit determines that it is to perform a write action. To carry outthis action, the target circuit writes (i.e., posts) a bus transactionvalue (step 1003) called a pull-id onto the pull-id mesh. The pull-id isalso of the format indicated in FIG. 18. The payload portion of thepull-id is of the format set forth in FIG. 20. The final destinationfield of the metadata portion of the pull-id indicates the island wherethe master circuit is located. The target port field identifies whichsub-circuit target it is within the target's island that is the targetcircuit of the command. The pull-id is communicated through the pull-idmesh back to the master circuit.

The master circuit receives the pull-id from the pull-id mesh and usesthe content of the data reference field of the pull-id to find the data.In the overall write operation, the master circuit knows the data it istrying to write into the target circuit. The data reference value thatis returned with the pull-id is used by the master circuit as a flag tomatch the returning pull-id with the write operation the master circuithad previously initiated.

The master circuit responds by sending (step 1004) the identified datato the target across one of the data meshes data0 or data1 as a “pull”data bus transaction value. The term “pull” means that the data of theoperation passes from the master to the target. The term “push” meansthat the data of the operation passes from the target to the master. Theformat of the “pull” data bus transaction value sent in this sending ofdata is also as indicated in FIG. 18. The format of the payload portionin the case of the payload being pull data is as set forth in FIG. 22.The first bit of the payload portion is asserted. This bit being adigital high indicates that the transaction is a data pull as opposed toa data push. The target circuit then receives (step 1005) the data pullbus transaction value across the data1 or data0 mesh. The target circuitwrites the content of the data field (the data field of FIG. 22) of thepull data payload portion into target memory at the appropriate locationindicated by the address field of the original write command.

FIG. 24 is a flowchart of a read operation method 2000 that might occuracross the configurable mesh CPP data bus. In a first step (step 2001),a master circuit in one of the islands uses its data bus interface tooutput (to “post”) a bus transaction value onto the command mesh bus ofthe configurable mesh CPP data bus. In this case, the bus transactionvalue is a read command to read data from a target circuit. The formatof the read command is as set forth in FIGS. 18 and 19. The read commandincludes a metadata portion and a payload portion. The metadata portionincludes the 6-bit final destination value that indicates the islandwhere the target is located. The action field of the payload portion ofthe read command indicates that the command is a read. The 14-bit datareference field is usable by the master circuit as a flag to associatedreturned data with the original read operation the master circuitpreviously initiated. The address field in the payload portion indicatesan address in the target where the data is to be obtained. The lengthfield indicates the amount of data.

The target receives the read command (step 2002) and examines thepayload portion of the command. From the action field of the commandpayload portion the target circuit determines that it is to perform aread action. To carry out this action, the target circuit uses theaddress field and the length field to obtain the data requested. Thetarget then pushes (step 2003) the obtained data back to the mastercircuit across data mesh data1 or data0. To push the data, the targetcircuit outputs a push bus transaction value onto the data1 or data0mesh. FIG. 23 sets forth the format of the payload portion of this pushbus transaction value. The first bit of the payload portion indicatesthat the bus transaction value is for a data push, as opposed to a datapull. The master circuit receives the bus transaction value of the datapush (step 2004) from the data mesh bus. The master circuit then usesthe data reference field of the push bus transaction value to associatethe incoming data with the original read command, and from the originalread command determines where the pushed data (data in the date field ofthe push bus transaction value) should be written into the mastercircuit. The master circuit then writes the content of the data field ofthe data field into the master's memory at the appropriate location.

As explained above, the contents of the LUTs of the crossbar switches ofthe configurable mesh data bus determine how bus transaction values willbe routed on their ways to their final destinations. The contents of theLUTs are set so that the configurable mesh data bus contains no loops.For a bus transaction value having a metadata portion indicating a givenfinal destination, and given that this bus transaction value is injectedonto a given link, there is only one path through the configurable meshdata bus that the bus transaction value can take to get to its finaldestination. Having only one path avoids deadlocks. Also, for a givenapplication to which the IB-NFP integrated circuit is to be put, theanticipated traffic from each source of traffic on the chip to eachdestination of traffic on the chip is determined. For the anticipatedtraffic over each such source-to-destination path, the routing throughthe data mesh is configured so that the anticipated traffic will bespread and balanced across the mesh data bus so that no link of the meshdata bus will be overburdened. The configuration information for theLUTs of the crossbar switches is loaded into the various LUTs using theconfigurable control bus (CB). The LUTs of the configurable mesh databus are loaded in this way initially at startup of the IB-NFP integratedcircuit before any communication across the configurable mesh data busoccurs.

FIG. 25 is a diagram of the configurable mesh control bus structure ofthe IB-NFP integrated circuit 12. As set forth above in connection withFIG. 8, the configurable mesh control bus structure of each full islandinvolves a centrally located switch, and four duplex half links. A firsthalf link extends from the switch up and to the left to a first portlocation P1 of the island. A second half link extends from the switch upand to the right to a second port location P2 of the island. A thirdhalf link extends from the switch down and to the right to a third portlocation P3 of the island. A fourth half link extends from the switchdown and to the left to a fourth port location P4 of the island. Theseport locations are marked on island 66. The control bus does not have tocommunicate high speed data in this example. Information passing acrossthe control bus from one switch of one island to another switch of anadjacent island does not pass through many registers. In other examples,links of the control bus may have pipelining registers in the routes oflinks between switches, where the pipelining registers are clocked bythe 1.0 GHz global clock.

Although the islands of IB-NFP integrated circuit 12 are illustrated asabutting one another, in some examples there is actually a smallinterstitial spacing between adjacent islands. FIG. 26 is a top-downdiagram of a part of the IB-NFP integrated circuit. FIG. 26 shows theinterstitial spacings between islands and shows how the configurablemesh data bus bridges those spacings. FIG. 27 is a top-down diagram ofthe same part of the IB-NFP integrated circuit. FIG. 27 shows theinterstitial spacings between islands and shows how the configurablecontrol bus bridges those spacings. FIG. 28 is a top-down diagram ofsame part of the IB-NFP integrated circuit. FIG. 28 shows theinterstitial spacings between islands and shows how the configurablemesh event bus bridges those spacings.

FIG. 29 is a top-down diagram of the configured configurable meshcontrol bus structure of the IB-NFP integrated circuit 12. Theconfigurable mesh control bus structure is configured to form a treestructure. The source or root of the tree structure is the functionalcircuitry in the ARM island 51. The arrows on the various half links ofthe control bus in the illustration of FIG. 29 indicate howconfiguration information flows through the tree structure.Configuration information flows from the ARM island 51 and down throughintermediary islands 53-58, 61-66, 72 and 73 of the branch structure andto the end leaf islands 50, 52, 59, 60, 67, 68, 70, 71, 74 of the treestructure, but configuration information cannot flow in the reversedirection toward the ARM island 51. Configuration information forloading the LUTs of the crossbar switches of the configurable mesh databus is stored in external configuration PROM 13. After power up of theIB-NFP integrated circuit 12, functional circuitry in the ARM island 51reads this configuration information out of external configuration PROM13 via GPIO interface block 48. The configuration information for agiven LUT is then communicated down through the whole tree. Theappropriate crossbar switch of the island of which the LUT is a partwill match a selection field portion of the configuration informationwith its hard-wired island number. Each island is provided with ahard-wire island number. The island uses the hard-wire island number toexamine information of the control bus and to determine whichconfiguration has an associated selection field portion destined for theisland number. When such a match occurs, the identified configurationinformation is loaded into the LUTs of the island. The configuration ofthe configurable control bus into the tree structure is staticallydetermined by hardwired tie off connections. The configuration of thecontrol bus is therefore set at the time of chip manufacture.

FIG. 30 is a simplified top-down diagram of the control bus portion ofARM island 51 and parts of ME island 53 and ME island 54. Each half linkhas an associated tie off input conductor that can be either tied to adigital logic high voltage or to a digital logic low voltage. If the tieoff conductor is tied to a digital high, then the associated input halflink portion is configured to receive control information into theisland through that port location. If the tie off conductor is tied to adigital low, then the associated input half link portion is configuredto output control information from the central switch. In the example ofFIG. 30, the tie offs for the input ports P1, P2, P3 and P4 for thecontrol bus of ARM island 51 are all tied low. The source of controlinformation is therefore the functional circuitry 165 of the island 51.Regardless of the source, the control information is output to each ofthe output ports of the island by the control bus half links. Thecontrol information is also supplied to the functional circuitry via theoutput port that couples to the functional circuitry 165. The controlinformation originates from configuration PROM 13, flows through GPIOinterface block 48, and through functional circuitry 165 of ARM island51, and through switch 166, and out to the four output ports P1, P2, P3and P4 (and also back to the functional circuitry 165). Ports P1 and P2are not coupled to another island so that fact that the configurationinformation flows out to these output ports is of no moment. Theconfiguration information that flows out to output port P4, however,flows into ME island 53 through input port P2 of island 53, through halflink portion 168, and to switch 167. The tie off associated with inputport P2 of ME island 53 is tied high. This configures port P2 of MEisland 53 to be the source of configuration information for the MEisland. At most one tie off of one port of an island can be tied high.If no tie off is tied high, then the source for the configurationinformation is the functional circuit of the island.

Likewise, the configuration information flows out of port P3 of ARMisland 51 and into input port P1 of ME island 54, through half linkportion 170, and to central switch 169. The tie off associated withinput port P1 of ME island 54 is tied high. This configures port P1 ofME island 54 to be the source of configuration information for the MEisland 54.

FIG. 31 is a diagram that shows how configuration information flows intoand through ME island 53. As described above in connection with FIG. 30,the configuration information flows into port P2 of ME island 53 and toswitch 167. The tie off for port P2 of island 53 is tied high and as aresult this input port is configured to be the source of theconfiguration information for the island. The configuration informationis output to all ports and to the functional circuitry of the island. Inthis way, the control bus structure of each island ishardwire-configured with tie offs such that the control bus structure ofthe overall IB-NFP integrated circuit 12 is configured to form the treestructure as illustrated in FIG. 29.

FIG. 32 is a simplified diagram of the central switch of the control busstructure of an island. The labels P1_IN, P2_IN, P3_IN and P4_INindicate the input half link portions of the island at input ports P1,P2, P3 and P4 that are possible sources of configuration information forthe island. Configuration information present at any one of these inputports, or on the output IOUT of the functional circuitry of the island,is multiplexed by multiplexer 176 onto all the output half link portionsof the island: P1_OUT, P2_OUT, P3_OUT, P4_OUT, and onto the input IIN ofthe functional circuitry of the island. Which one of the input ports (orthe functional circuitry) it is that will be the source of theconfiguration information is determined by the tie off signal valuesSEL_P1, SEL_P2, SEL_P3 and SEL_P4. SEL P1 is the tie off value at inputport P1. The tie off signal SEL P2 is the tie off value at input portP2. The tie off signal SEL P3 is the tie off value at input port P3. Thetie off signal SEL P4 is the tie off value at input port P4. Each fullisland in the IB-NFP integrated circuit 12 has four such tie offs. Ifnone of the tie offs is tied high, then the source of configurationinformation is the IOUT from the functional circuitry of the island. Inone example, power and ground conductors extending in upper layer metalextend through the interstitial space between adjacent islands, and eachtie off tab conductor of the island is coupled up to one of theseconductors by an appropriate vertically extending conductive via. Thetie off tab conductors are illustrated in FIG. 31 as rectangular tabstructures 172-175. The round circle seen in each of the tab conductorsrepresents the vertically-extending conductive via that couples the tabconductor to either a power conductor or a ground conductor in the metallayers above.

FIG. 33 is a configurable mesh event bus structure of the IB-NFPintegrated circuit 12. From the top-down view shown in FIG. 33 four portlocations are visible on the top overlay layer; however, each islandalso has a half link portion extending down into each island couplingthe Real Time Configurable Switch (RTCS) to functional circuitry withinthe island. For example, ingress NBI island 72 is an island with fourhalf link portions coupling four port locations (P1-P4) on the topoverlay layer to the RTCS; however, ingress NBI island 72 also has afifth half link portion coupling the RTCS to functional circuitry withinthe ingress NBI island 72. As set forth above in connection with FIG. 9,the configurable mesh event bus structure of each full island involves acentrally located RTCS, and five duplex half links. A first half linkextends from the RTCS up and to the left to a first port location P1 ofthe island. A second half link extends from the RTCS up and to the rightto a second port location P2 of the island. A third half link extendsfrom the RTCS down and to the right to a third port location P3 of theisland. A fourth half link extends from the RTCS down and to the left toa fourth port location P4 of the island. A fifth half link extends fromthe RTCS down to the functional circuitry within the island. These portlocations are marked on ingress NBI island 72 in FIG. 33.

FIG. 34 is a perspective view of the local event ring within NBI island72. As discussed above, the ingress NBI island 72 includes four halflinks (201-204) and a centrally located Real Time Configurable Switch(RTCS) 206. RTCS 206 may also be referred to as a “switch”. Half link201 includes a half link portion P1_IN that can communicate informationfrom port location P1 to RTCS 206 and a half link portion P1_OUT thatcan communicate information from switch 206 to port location P1. Halflink 202 includes a half link portion P2_IN that can communicateinformation from port location P2 to RTCS 206 and a half link portionP2_OUT that can communicate information from RTCS 206 to port locationP2. Half link 203 includes a half link portion P3_IN that cancommunicate information from port location P3 to RTCS 206 and a halflink portion P3_OUT that can communicate information from RTCS 206 toport location P3. Half link 204 includes a half link portion P4_IN thatcan communicate information from port location P4 to RTCS 206 and a halflink portion P4_OUT that can communicate information from RTCS 206 toport location P4. Link 205 includes a link portion P5_IN that cancommunicate information from functional circuitry 207 within the ingressNBI island 72 to RTCS 206 and a link portion P5_OUT that can communicateinformation from RTCS 206 to functional circuitry 207 within the NBIisland 72. A RTCS control logic 208 is coupled to RTCS 206. The RTCScontrol logic 208 includes instructions associated with each half linkportion coupled to RTCS 206. The RTCS control logic 208 controls whichinputs to RTCS 206 are coupled to which outputs of RTCS 206. The RTCScontrol logic 208 is programmed via the configurable mesh control bus(CB). The configurable mesh control bus (CB) is coupled to the RTCScontrol logic 208. In one embodiment, the RTCS control logic 208 in eachisland is programmed once via the configurable mesh control bus uponpowering on the IB-NFP integrated circuit 12. In another, embodiment,the RTCS control logic 208 in each island is programmed not only uponpowering on the IB-NFP integrated circuit 12, but also at another timeduring subsequent operation of the IB-NFP integrated circuit 12.

Functional circuitry 207 includes a first event ring circuit 209 and asecond event ring circuit 210. Link portion P5_OUT couples to an inputterminal of event ring circuit 209 via a first event ring segment. Anoutput terminal of event ring circuit 209 couples to an input terminalof event ring circuit 210 via a second event ring segment. An output ofevent ring circuit 210 couples to link portion P5_IN via a third eventring segment. Event ring circuit 209 is described in FIG. 36. Event ringcircuit 210 is described in FIG. 37. The event packet 200 is furtherdescribed in FIGS. 38A and 38B. Vacancy indicator bit 239, source ofevent 240, and type of event 241 form a sort of header of the eventpacket, and event data 242 is a payload of the event packet.

In operation, an event packet 200 is communicated to RTCS 206 via aninput half link. RTCS 206 then communicates the event packet 200 viaanother half link portion according to the instructions stored in RTCScontrol logic 208. For example, event packet 200 from the ingress ME 66may be communicated via half link portion P2_IN to RTCS 206 asillustrated in FIG. 34. RTCS 206 then may communicate the event packet200 via another half link portion to any one of the four port locationsor to the functional circuitry 207 within the NBI island 72. The RTCS206 is configured, via RTCS control logic 208, to couple half linkportion P2_IN to link portion P5_OUT. The RTCS 206 is also configured tocouple link portion P5_IN to half link portion P1_OUT. The event packet200 is communicated from port location P2 to RTCS 206 via half linkportion P2_IN. The event packet 200 is communicated from RTCS 206 tofunctional circuitry 207 via link portion P5_OUT. The event packet 200is then communicated from link portion P5_OUT to the input terminal ofevent ring circuit 209 via an event ring segment. The event packet 200is then communicated from the output terminal of event ring circuit 209to the input terminal of event ring circuit 210 via an event ringsegment. The event packet 200 is then communicated from the outputterminal of event ring circuit 210 to link portion P5_IN via an eventring segment. The event packet 200 is then communicated to RTCS 206 vialink portion P5_IN. The event packet 200 is then communicated to portlocation P1 from RTCS 206 via half link portion P1_OUT. The event packet200 is then communicated through port location P1 to Interlaken LA(1)island 65.

Alternatively, RTCS 206, via RTCS control logic 208, may be configuredsuch that half link portion P1_IN is coupled directly to half linkportion P2_OUT. Therefore, event packet 200 from ME island 66 would notbe communicated to functional circuitry 207, but rather would bedirectly communicated to port location P1 and subsequently InterlakenLA(1) island 65. This alternative may be used to connect a local eventring through one island without making the functional circuitry of thesaid one island part of the local event ring.

A global clock signal is not shown in FIG. 34; however, a global clocksignal is coupled to RTCS 206, event ring circuit 209, and event ringcircuit 210. The global clock signal may be a variety of differentfrequencies. In one embodiment, the global clock is a one gigahertzsignal and is coupled to all local event ring circuitry. Depending onthe frequency of global clock signal intermediate registers, or “slots”,may be required to address propagation delay in sending event packetsbetween various event ring circuits. In an embodiment, a slot may becoupled in series at multiple locations along the local event ring. Forexample, one or more slots may be coupled in series along a half linkportion or between local event ring circuits.

FIG. 35 is a functional diagram of the RTCS 206 shown in FIG. 34. RTCS206 includes five multiplexers (211-215) and RTCS control logic 208.RTCS 206 has five inputs P1_IN, P2_IN, P3_IN, P4_IN and P5_IN. RTCS 206has five outputs P1_OUT, P2_OUT, P3_OUT, P4_OUT, and P5_OUT. Each outputhas an independent multiplexer (211-215), which couples one of the fiveinputs to a single output terminal of each multiplexer. Each multiplexeris controlled by an individual set of select control lines (SEL1-SEL15).In one embodiment, each multiplexer has three select control lines.Three controls lines results in the ability to control up to eightpossible multiplexer inputs. RTCS control logic 208 has a clock input(CLK), an enable input (EN), a configurable mesh control bus input, andfifteen select control outputs (SEL 1-15). Configuration information iscommunicated via the configurable mesh control bus to RTCS control logic208. RTCS control logic 208 sources a signal on select control outputs(SEL 1-15) based upon the configuration information. Three selectcontrol lines (SEL 1-3) are coupled to multiplexer 211. Three selectcontrol lines (SEL 4-6) are coupled to multiplexer 212. Three selectcontrol lines (SEL 7-9) are coupled to multiplexer 213. Three selectcontrol lines (SEL 10-12) are coupled to multiplexer 214. Three selectcontrol lines (SEL 13-15) are coupled to multiplexer 215. The threesignals communicated to a multiplexer and control which, if any, of themultiplexer inputs are coupled to the multiplexer output. In oneembodiment, the signal on a select control line may be a logic low orhigh signal, meaning that the voltage applied to the control line beabove or below a threshold voltage.

In operation, configuration information is communicated to RTCS controllogic 208 via the configurable mesh control bus. According to theconfiguration information received, RTCS control logic 208 sources anindependent signal on each select control line (SEL1-15). The signalsare coupled to respective multiplexers as described above. In responseto the signal received via select control lines, each multiplexercouples one of its inputs to the output terminal of the multiplexer orleaves the output terminal in a floating state.

FIG. 36 is an expanded diagram of event ring circuit 209 shown in FIG.34. Event ring circuit 209 includes register 217, multiplexer 218,vacancy logic 219, event packet generating circuit 220, First In FirstOut circuit (FIFO) 221, source number checker 222, and multiplexer 223.A previous local event ring segment is coupled to the data inputterminal of register 217. A global clock signal 254 is coupled to theclock input terminal of register 217. The output terminal of register217 couples to a first input terminal of multiplexer 223, to an eventpacket input terminal (EP) of source number checker 222 and to an inputterminal of vacancy logic 219. A second input terminal of multiplexer223 is coupled to a logic high signal, such as a voltage source. Anoutput terminal of multiplexer 223 is coupled to a first input terminalof multiplexer 218. An output terminal of vacancy logic 219 is coupledto a selector input terminal of multiplexer 218. The output terminal ofvacancy logic 219 is also coupled to a present input terminal (PR) ofFIFO 221. A global clock signal 254 is coupled to the clock inputterminal of source number checker 222. The configurable mesh control busis coupled to a control bus input terminal (CB) of source number checker222. A control output terminal (CN) of source number checker 222 iscoupled to a selector input terminal of multiplexer 223. A source numberoutput terminal (SN) of source number checker 222 is coupled to a sourcenumber (SN) input terminal of event packet generating circuit 220. Aglobal clock signal 254 is coupled to the clock input terminal of eventpacket generating circuit 220. An event packet output terminal (EP) ofevent packet generating circuit 220 is coupled to a data input terminalof FIFO circuit 221. A push output terminal (P) of event packetgenerating circuit 220 is coupled to a push input terminal (P) of FIFO221. A full (F) output control terminal of FIFO 221 is coupled to a full(F) input terminal of event circuit generating circuit 220. A dataoutput terminal of FIFO circuit 221 is coupled to a second inputterminal of multiplexer 218. An output terminal of multiplexer 218 iscoupled to the next segment of the event ring.

In operation, event packet generating logic 220 creates event packets tobe inserted into a local event ring. Event packets created by eventpacket generating circuit 220 represent events which occurred within theisland where the event ring circuit is located. In the embodimentillustrated in FIG. 36, event packet generating circuit 220 createsevent packets representing events which occurred in NBI island 72. Theevent packet 225 created by event packet generating logic 220 iscommunicated to FIFIO 221. FIFO 221 acts as a buffer to store eventpackets until the event packets can be inserted into the local eventring. Event packets can only be inserted into the local event ring whenthe previous event ring segment is carrying no event packet. The eventpacket generating circuit 220 communicates a signal from the push outputterminal (P) of the event packet generating circuit 220 to the pushinput terminal (P) of FIFO 221 when the event packet generating circuit220 is communicating an event packet from the event packet outputterminal (EP) to the data input terminal of FIFO 221. FIFO 221 has afinite memory size. In the event that FIFO 221 memory is full, noadditional event packets from event packet generating circuit 220 may bestored in FIFO 221. A signal indicating that the FIFO 221 is full iscommunicated from the full output terminal (F) of FIFO 221 to the fullinput terminal (F) of event packet generating circuit 220. FIFO 221always outputs a value; however, if no event packets are stored in FIFO221 an output value with a first bit set to one (1) is communicated tothe second terminal of multiplexer 218.

A first value is clocked into register 217 from the previous event ringsegment. The first value is then communicated from the output ofregister 217 to the input terminal of vacancy logic 219, the eventpacket input terminal (EP) of source number checker 222, and the firstinput terminal of multiplexer 223.

Vacancy logic 219 receives the first bit of the first value communicatedfrom register 217. The first bit of the first value indicates if anevent packet it present on the segment. The first bit is one (1) if theevent ring segment is carrying no event packet. The first bit of thefirst value is zero (0) if an event packet is present on the event ringsegment.

Source number checker 222 receives an assigned source number from theconfigurable mesh control bus via the control bus input terminal (CB).The assigned source number is a number that is unique to event ringcircuit 209. All event packets inserted into a local event ring by eventring circuit 209 will have a “source of event” field that indicates thesource number assigned to event ring circuit 209. In one embodiment, the“source of event” filed is eight bits wide. An event packet bit sequencemap and table is provided in FIGS. 38A and 38B. Source number checker222 communicates the assigned source number from the source numberoutput terminal (SN) of source number checker 222 to the source numberinput terminal (SN) of event packet generating circuit 220. Event packetgenerating circuit 220 uses the source number information received fromsource number checker 222 to fill in the “source of event” field in newevent packets.

Source number checker 222 also receives eight bits of the first valuevia the event packet input terminal (EP). The eight bits make up the“source of event” field and indicate which event ring circuit in thelocal event ring inserted the value. The source number checker 222compares the received eight bits with the eight bit source numberassigned to event ring circuit 209 via the configurable mesh controlbus. When the eight bits representing the assigned source number are thesame as the received eight bits of the first value, the source numberchecker 222 communicates a signal from control output terminal (CN) tothe selector input terminal of multiplexer 223, such that multiplexer223 couples the second input terminal of multiplexer 223 to the outputterminal of multiplexer 223. The result is that source number checker222 couples the logic high signal to the output terminal of multiplexer223 when the eight source of event bits match, and that the sourcenumber checker 222 couples the output of register 217 to the output ofmultiplexer 223 when the eight source of event bits do not match.Coupling the logic high signal to the output terminal of multiplexer 223has the effect of passing no event packet to the first input terminal ofmultiplexer 218. Source number checker 222 also receives a global clocksignal 254 via the clock input terminal. The source number checker 222transmits and receives communications during transitions of the globalclock signal.

Vacancy logic 219 receives the first bit of the first value communicatedfrom register 217. The first bit of the first value is one (1) if noevent packet is present. The first bit of the first value is zero (0) ifan event packet is present.

When an event packet 224 is present, the first bit of the first value isa zero (0) and vacancy logic 219 will communicate a control signal tothe selector input terminal of multiplexer 218 so that multiplexer 218will couple the first input terminal of multiplexer 218 to the outputterminal of multiplexer 218. The vacancy logic 219 will also communicatea control signal to the present input terminal (PR) of FIFO 221 so thatFIFO 221 will not communicate an event packet to the second inputterminal of multiplexer 218. Therefore, when the first bit of the firstsignal is zero (0) either the event packet 224 or the logic high signalwill be coupled to the next event ring segment. Whether the event packet224 or the logic high signal is coupled to the next event ring segmentdepends on the signal communicated from the control output terminal (CN)of source number checker 222. If the first bit of the first value iszero (0) and the received source number does not match the assignedsource number, then event packet 224 will be coupled to the next eventring segment. If the first bit of the first value is zero (0) and thereceived source number does match the assigned source number, then thelogic high signal will be coupled to the next event ring segment.Coupling the logic high signal to the next event ring segment has theeffect of deleting event packet 224 from the local event ring andinserting no event packet in the local event ring.

When, on the other hand, an event packet is not present, the first bitof the first value is one (1) and vacancy logic 219 will communicate acontrol signal to the selector input terminal of multiplexer 218 so thatmultiplexer 218 will couple the second input terminal of multiplexer 218to the output terminal of multiplexer 218. The vacancy logic 219 willalso communicate a control signal to the present input terminal (PR) ofFIFO 221 so that FIFO 221 will communicate event packet 226 to thesecond input terminal of multiplexer 218. Therefore, when the first bitof the first value is a one (1) the next event packet 226 stored in FIFO221 will be coupled to the next event ring segment.

The configurable mesh control bus is coupled to the control bus inputterminal (CB) of source number checker 222. Source number checker 222may be assigned a source number via the configurable mesh control bus.In one embodiment, source number checker 222 is assigned a source numberonce upon powering on the IB-NFP integrated circuit 12. In anotherembodiment, source number checker 222 is assigned a first source numberupon powering on the IB-NFP integrated circuit 12 and a second sourcenumber, replacing the first source number, during subsequent operationof the IB-NFP integrated circuit 12.

FIG. 37 is an expanded diagram of event ring circuit 210 shown in FIG.34. Event ring circuit 210 includes a global event filter 227 inaddition to the items included in event ring circuit 209. Event ringcircuit 210 includes register 270, multiplexer 271, vacancy logic 272,event packet generating circuit 273, First In First Out circuit (FIFO)274, source number checker 275, multiplexer 276, and global event filter227. Global event filter 227 includes register 228, source numberchecker 229, vacancy logic 230, multiplexer 231, multiplexer 232, globalevent filter logic 233, and FIFO 234.

A previous local event ring segment is coupled to the data inputterminal of register 270. A global clock signal 254 is coupled to theclock input terminal of register 270. The output terminal of register270 couples to a first input terminal of multiplexer 276, to an eventpacket input terminal (EP) of source number checker 275 and to an inputterminal of vacancy logic 272. A second input terminal of multiplexer276 is coupled to a logic high signal, such as a voltage source. Anoutput terminal of multiplexer 276 is coupled to a first input terminalof multiplexer 271. An output terminal of vacancy logic 272 couples to aselector input terminal of multiplexer 271. The output terminal ofvacancy logic 272 also couples to a present input terminal (PR) of FIFO274. A global clock signal 254 is coupled to the clock input terminal ofsource number checker 275. The configurable mesh control bus is coupledto a control bus input terminal (CB) of source number checker 275. Acontrol output terminal (CN) of source number checker 275 is coupled toa selector input terminal of multiplexer 276. A source number outputterminal (SN) of source number checker 275 is coupled to a source number(SN) input terminal of event packet generating circuit 273. A globalclock signal 254 is coupled to the clock input terminal of event packetgenerating circuit 273. An event packet output terminal (EP) of eventpacket generating circuit 273 is coupled to a data input terminal ofFIFO circuit 274. A push output terminal (P) of event packet generatingcircuit 273 is coupled to a push input terminal (P) of FIFO 274. A full(F) output control terminal of FIFO 274 is coupled to a full (F) inputterminal of event circuit generating circuit 273. A data output terminalof FIFO circuit 274 is coupled to a second input terminal of multiplexer271. An output terminal of multiplexer 271 is coupled to the nextsegment of the event ring.

Within global filter 227, a previous global event chain segment iscoupled to the data input terminal of register 228. The output terminalof register 228 is coupled to a first input terminal of multiplexer 231,an event packet input terminal (EP) of source number checker 229, and aninput terminal of vacancy logic 230. A global clock signal 254 iscoupled to the clock input terminal of register 228. A second inputterminal of multiplexer 231 is coupled to a logic high signal, such as avoltage source. An output terminal of multiplexer 231 is coupled to asecond input of multiplexer 232. An output terminal of vacancy logic 230couples to a selector input terminal of multiplexer 232. A global clocksignal 254 is coupled to the clock input of source number checker 229.The configurable mesh control bus is coupled to the control bus inputterminal (CB) of source number checker 229. A control output terminal(CN) of source number checker 229 is coupled to a selector inputterminal of multiplexer 231. A source number output terminal (SN) ofsource number checker 229 is coupled to a source number input terminal(SN) of global event filter 233. A global clock signal 254 is coupled tothe clock input terminal of global event filter logic 233.

The output terminal of register 217 is coupled to the event packet inputterminal (EP) of global event filter logic 233. Configurable meshcontrol bus is coupled to a control bus input terminal (CB) of globalevent filter 223. A global event output terminal (GE) of global eventfilter logic 223 is coupled to a data input terminal of FIFO circuit234. A push output terminal (P) of global event filter logic 233 iscoupled to a push input terminal (P) of FIFO 221. A full output terminal(F) of FIFO 234 is coupled to a full input terminal (F) of global eventfilter logic 233. A data output terminal of FIFO circuit 234 is coupledto a first input terminal of multiplexer 232. An output terminal ofmultiplexer 232 is coupled to the next segment of the global eventchain.

In operation with respect to local event ring, event packet generatinglogic 273 creates event packets to be inserted into a local event ring.Event packets created by event packet generating circuit 273 representevents which occurred within the island where the event ring circuit islocated. In the embodiment illustrated in FIG. 36, event packetgenerating circuit 273 creates event packets representing events whichoccurred in NBI island 72. The event packet 278 created by event packetgenerating logic 273 is communicated to FIFIO 274. FIFO 274 acts as abuffer to store event packets until the event packets can be insertedinto the local event ring. Event packets can only be inserted into thelocal event ring when the previous event ring segment is carrying noevent packet. The event packet generating circuit 273 communicates asignal from the push output terminal (P) of the event packet generatingcircuit 273 to the push input terminal (P) of FIFO 274 when the eventpacket generating circuit 273 is communicating an event packet from theevent packet output terminal (EP) to the data input terminal of FIFO274. FIFO 274 has a finite memory size. In the event that FIFO 274memory is full, no additional event packets from event packet generatingcircuit 273 may be stored in FIFO 274. A signal indicating that the FIFO274 is full is communicated from the full output terminal (F) of FIFO274 to the full input terminal (F) of event packet generating circuit273. FIFO 274 always outputs a value; however, if no event packets arestored in FIFO 274 an output value with a first bit set to one (1) iscommunicated to the second terminal of multiplexer 271. A first value isclocked into register 270 from the previous event ring segment during atransitioning of global clock signal 254. The first value is thencommunicated from the output terminal of register 270 to the inputterminal of vacancy logic 272, the event packet input terminal (EP) ofsource number checker 275, and the first input terminal of multiplexer276.

Source number checker 275 receives an assigned source number from theconfigurable mesh control bus via the control bus input terminal (CB).The assigned source number is a number that is unique to event ringcircuit 210. All event packets inserted into a local event ring by eventring circuit 210 will have a “source of event” field that indicates thesource number assigned to event ring circuit 210. In one embodiment, the“source of event” filed is eight bits wide. An event packet bit sequencemap and table is provided in FIGS. 38A and 38B. Source number checker275 communicates the assigned source number from the source numberoutput terminal (SN) of source number checker 275 to the source numberinput terminal (SN) of event packet generating circuit 273. Event packetgenerating circuit 273 uses the source number information received fromsource number checker 275 to fill in the “source of event” field in newevent packets.

Source number checker 275 also receives eight bits of the first valuevia the event packet input terminal (EP). The eight bits make up the“source of event” field and indicate which event ring circuit in thelocal event ring inserted the value. The source number checker 275compares the received eight bits with the eight bit source numberassigned to event ring circuit 210 via the configurable mesh controlbus. When the eight bits representing the assigned source number are thesame as the received eight bits of the first value, the source numberchecker 275 communicates a signal from control output terminal (CN) tothe selector input terminal of multiplexer 276, such that multiplexer276 couples the second input terminal of multiplexer 276 to the outputterminal of multiplexer 276. The result is that source number checker275 couples the logic high signal to the output terminal of multiplexer276 when the eight source of event bits match, and that the sourcenumber checker 275 couples the output of register 270 to the output ofmultiplexer 276 when the eight source of event bits do not match.Coupling the logic high signal to the output terminal of multiplexer 276has the effect of passing no event packet to the first input terminal ofmultiplexer 271. Source number checker 275 also receives a global clocksignal 254 via the clock input terminal. The source number checker 275transmits and receives communications during transitions of the globalclock signal.

Vacancy logic 272 receives the first bit of the first value communicatedfrom register 270. The first bit of the first value is one (1) if noevent packet is present. The first bit of the first value is zero (0) ifan event packet is present.

When an event packet 277 is present, the first bit of the first value isa zero (0) and vacancy logic 272 will communicate a control signal tothe selector input terminal of multiplexer 271 so that multiplexer 271will couple the first input terminal of multiplexer 271 to the outputterminal of multiplexer 271. The vacancy logic 272 will also communicatea control signal to the present input terminal (PR) of FIFO 274 so thatFIFO 274 will not communicate an event packet to the second inputterminal of multiplexer 271. Therefore, when the first bit of the firstvalue is zero (0) either the event packet 277 or the logic high signalwill be coupled to the next event ring segment. Whether the event packet277 or the logic high signal is coupled to the next event ring segmentdepends on the signal communicated from the control output terminal (CN)of source number checker 275. If the first bit of the first value iszero (0) and the received source number does not match the assignedsource number, then event packet 277 will be coupled to the next eventring segment. If the first bit of the first value is zero (0) and thereceived source number does match the assigned source number, then thelogic high signal will be coupled to the next event ring segment.Coupling the logic high signal to the next event ring segment has theeffect of deleting event packet 277 from the local event ring andinserting no event packet in the local event ring.

When an event packet is not present, the first bit of the first value isone (1) and vacancy logic 272 will communicate a control signal to theselector input terminal of multiplexer 271 so that multiplexer 271 willcouple the second input terminal of multiplexer 271 to the outputterminal of multiplexer 271. The vacancy logic 272 will also communicatea control signal to the present input terminal (PR) of FIFO 274 so thatFIFO 274 will communicate event packet 279 to the second input terminalof multiplexer 271. Therefore, when the first bit of the first value isa one (1) the next event packet 279 stored in FIFO 274 will be coupledto the next event ring segment.

In operation with respect to the global event chain, global event filterlogic 233 filters monitors event packet being communicated along thelocal event ring. All event packets have a “type of event” fieldindicating the category of event the event packet is representing. Anevent packet bit sequence map and table is provided in FIGS. 38A and38B. The global event filter logic 233 compares the “type of event” bitsof an event packet being communicated along the local event ring to anarray of “type of event” bit sequences that have been identified asrepresenting global events. In one embodiment, the array of “type ofevent” bit sequences is communicated to the global event filter logic233 via the configurable mesh control bus. In another embodiment, thearray of “type of event” bit sequences is hardcoded within the globalevent circuit logic 233. In an embodiment, the “type of event” fieldwithin the event packet is four bits wide.

In the embodiment illustrated in FIG. 37, event packet 277 iscommunicated from the output terminal of register 270 to the eventpacket input terminal (EP) of global event filter logic 233. When eventpacket 277 has a “type of event” bit sequence that is the same as one ofthe array of “type of event” bit sequences identified as representingglobal event, global event filter logic 233 communicates event packet235 from the global event output terminal (GE) of global event filterlogic 233 to the data input terminal of FIFO 234. In one embodiment,event packet 235 is identical to event packet 277. In anotherembodiment, the “vacancy indicator”, “type of event”, and “event data”fields of event packet 235 are the same as those of event packet 277;however, the “source of event” field of event packet 235 is updated withthe source number assigned to event ring circuit 210. FIFO 234 acts as abuffer to store event packets until the event packets can be insertedinto the global event ring. Event packets can only be inserted into theglobal event ring when the previous event chain segment is carrying noevent packet. The global event filter logic 233 communicates a signalfrom the push output terminal (P) of the global event filter logic 233to the push input terminal (P) of FIFO 234 when the global event filterlogic 233 is communicating an event packet from the global event outputterminal (GE) to the data input terminal of FIFO 234. FIFO 234 has afinite memory size. In the event that FIFO 234 memory is full, noadditional event packets from global event filter logic 233 may bestored in FIFO 234. A signal indicating that the FIFO 234 is full iscommunicated from the full output terminal (F) of FIFO 234 to the fullinput terminal (F) of global event filter logic 233. FIFO 234 alwaysoutputs a value; however, if no event packets are stored in FIFO 234 anoutput value with a first bit set to one (1) is communicated to thefirst terminal of multiplexer 232.

A second value is clocked into register 228 from the previous eventchain segment during a transitioning of global clock signal 254. Thesecond value is then communicated from the output terminal of register228 to the input terminal of vacancy logic 230, the event packet inputterminal (EP) of source number checker 229, and the first input terminalof multiplexer 231.

Source number checker 229 receives an assigned source number from theconfigurable mesh control bus via the control bus input terminal (CB).The assigned source number is a number that is unique to event ringcircuit 210. In one embodiment, all event packets inserted into a globalevent chain by event ring circuit 210 will have a “source of event”field that indicates the source number assigned to event ring circuit210. In another embodiment, event packets inserted into a global eventchain by event ring circuit 210 will have a “source of event” field thatindicates a source number assigned to one event ring circuit within thesame local event ring as event ring circuit 210. In one embodiment, the“source of event” field is eight bits wide. An event packet bit sequencemap and table is provided in FIGS. 38A and 38B. Source number checker229 communicates the assigned source number from the source numberoutput terminal (SN) of source number checker 229 to the source numberinput terminal (SN) of global event filter logic 233. Global eventfilter logic 233 may use the source number information received fromsource number checker 229 to fill in the “source of event” field inevent packets inserted into the global event chain.

Source number checker 229 also receives eight bits of the second valuevia the event packet input terminal (EP). The eight bits make up the“source of event” field and indicate which event ring circuit in theglobal event chain inserted the value. The source number checker 229compares the received eight bits with the eight bit source numberassigned to event ring circuit 210 via the configurable mesh controlbus. When the eight bits representing the assigned source number are thesame as the received eight bits of the first value, the source numberchecker 229 communicates a signal from control output terminal (CN) tothe selector input terminal of multiplexer 231, such that multiplexer231 couples the second input terminal of multiplexer 231 to the outputterminal of multiplexer 231. The result is that source number checker229 causes the coupling of the logic high signal to the output terminalof multiplexer 231 when the eight source of event bits match, and thatthe source number checker 229 causes the coupling of the output ofregister 228 to the output of multiplexer 231 when the eight source ofevent bits do not match. Coupling the logic high signal to the outputterminal of multiplexer 231 has the effect of passing no event packet tothe second input terminal of multiplexer 232. Source number checker 229also receives a global clock signal 254 via the clock input terminal.The source number checker 229 transmits and receives communicationsduring transitions of the global clock signal.

Vacancy logic 230 receives the first bit of the second valuecommunicated from register 228. The first bit of the second value is one(1) if no event packet is present. The first bit of the second value iszero (0) if an event packet is present.

When an event packet 256 is present, the first bit of the second valueis a zero (0) and vacancy logic 230 will communicate a control signal tothe selector input terminal of multiplexer 232 so that multiplexer 232will couple the second input terminal of multiplexer 232 to the outputterminal of multiplexer 232. The vacancy logic 230 will also communicatea control signal to the present input terminal (PR) of FIFO 234 so thatFIFO 234 will not communicate an event packet to the first inputterminal of multiplexer 232. Therefore, when the first bit of the secondvalue is zero (0) either the event packet 256 or the logic high signalwill be coupled to the next event chain segment. Whether the eventpacket 256 or the logic high signal is coupled to the next event chainsegment depends on the signal communicated from the control outputterminal (CN) of source number checker 229. If the first bit of thesecond value is zero (0) and the received source number does not matchthe assigned source number, then event packet 256 will be coupled to thenext event chain segment. If the first bit of the second value is zero(0) and the received source number does match the assigned sourcenumber, then the logic high signal will be coupled to the next eventchain segment. Coupling the logic high signal to the next event chainsegment has the effect of deleting event packet 256 from the globalevent chain and inserting no event packet in the global event chain.

When an event packet is not present, the first bit of the second valueis one (1) and vacancy logic 230 will communicate a control signal tothe selector input terminal of multiplexer 232 so that multiplexer 232will couple the first input terminal of multiplexer 232 to the outputterminal of multiplexer 232. The vacancy logic 230 will also communicatea control signal to the present input terminal (PR) of FIFO 234 so thatFIFO 234 will communicate event packet 236 to the first input terminalof multiplexer 232. Therefore, when the first bit of the second value isa one (1) the next event packet 236 stored in FIFO 234 will be coupledto the next event chain segment.

The configurable mesh control bus is coupled to the control bus inputterminal (CB) of source number checkers 222 and 229. Source numbercheckers 222 and 229 may be assigned a source number via theconfigurable mesh control bus. In one embodiment, source number checkers222 and 229 are assigned a source number once upon powering on theIB-NFP integrated circuit 12. In another embodiment, source numbercheckers 222 and 229 are assigned a first source number upon powering onthe IB-NFP integrated circuit 12 and a second source number, replacingthe first source number, during subsequent operation of the IB-NFPintegrated circuit 12.

FIG. 39 is a diagram of the configurable mesh event bus configured toform two local event rings and a global event chain. FIG. 39 shows theconfigurable mesh event bus of FIG. 33 configured to form a first 245and a second 246 local event ring and a single global event chain 247.Ingress NBI island 72 is the ingress NBI island 72 shown in FIGS. 33 and34. The first local event ring 245 flows through the ingress MAC island71, ingress NBI island 72, ingress ME island 66, ARM island 51 and eightother islands (53, 57, 62, 65, 60, 55, 61, and 56). The second localevent ring 246 flows through egress NBI island 63, egress MAC island 64and four other islands (54, 58, 59, and 69). The global event chain 247begins on ingress NBI island 72, couples through island 66, island 62,island 57, island 54 and terminates on ARM island 51. Both local eventrings 245 and 246 are unidirectional, in that event packets travel in asingle direction along the local event ring. The global event chain isunidirectional, in that global event packets travel in a singledirection along the global event chain. Whereas the global even chainhas a single point of termination on ARM island 51, both local eventrings 245 and 246 have no fixed termination location. Rather, eventpackets are removed at different points along the local event ringdepending upon where in the local event ring the event packet wasinserted into the local event ring. Each local event ring operates as asource-release ring.

FIG. 40 is a simplified system level illustration of a source-releasering. The source-release ring includes event ring circuits and eventring segments. An event ring segment couples a first event ring circuitto a second event ring circuit. The source-release ring illustrates theflow of the first local event ring 245 through ingress ME island 66,ingress NBI island 72, and Interlaken LA(1) island 65. The first localevent ring 245 flows through a single event ring circuit 248 within MEisland 66. The first local event ring 245 continues through the first209 and second 210 event ring circuits of ingress NBI island 72 to eventring circuit 249 within Interlaken LA(1) island 65. A source-releasering is a circuit in which an event packet is clocked into sequentialregisters along a single direction. The event packet travels around thering, stepping through the ring one register at a time. All eventpackets are transferred from the present register to the next registerupon the transition of a synchronized clock cycle. In a source-releasering, only the source of an event packet can delete the event packetfrom the ring. Event ring circuits must monitor the passing event packetand only delete the event packet if the present event ring circuit wasthe source of the event packet. FIG. 40 is a simplified drawing. Eventring segments may comprise switches, slots, traces on silicon or otherfunctional circuitry.

Referring to FIG. 39 and traveling clockwise around the source-releasering shown in FIG. 40, an event packet 280 is inserted into local eventring 245 by event ring circuit 248 within ME island 66. The event packet280 is then communicated via an event ring segment to the first eventring circuit 209 within ingress NBI island 72. Since an event packetgenerated by another event ring circuit is present, the first event ringcircuit 209 within ingress NBI island 72 cannot insert or delete anevent packet, but can monitor the contents of the event packet 280. Theevent packet 280 is then communicated via an event ring segment to thesecond event ring circuit 210 within ingress NBI island 72. Since anevent packet generated by another event ring circuit is present, thesecond event ring circuit 210 within ingress NBI island 72 cannot insertor delete an event packet, but can monitor the contents of the eventpacket 280. The event packet 280 is then communicated via an event ringsegment to an event ring circuit 249 within the Interlaken LA(1) island65. Since an event packet generated by another event ring circuit ispresent, the event ring circuit 249 within the Interlaken LA(1) island65 cannot insert or delete an event packet, but can monitor the contentsof the event packet 280. The event packet 280 will continue to becommunicated around the first local event ring 245 until it returns tothe event ring circuit 248 within the ingress ME island 66, where eventring circuit 248 will delete or “release” the event packet 280 from thelocal event ring 245.

FIG. 41 is a simplified diagram of the first 245 and second 246 localevent rings and single global event chain 247 shown in FIG. 39. Thefirst local event ring 245 includes an event ring circuit 248 withingress ME island 66, the first event ring circuit 209 within ingressNBI island 72, the second event ring circuit 210 within NBI island 72,an event ring circuit 249 within Interlaken LA(1) island 65, andmultiple event ring segments coupling each event ring circuit to thefollowing event ring circuit. The second local event ring 246 includesevent ring circuit 250 within ME cluster island 54, other event ringcircuits and multiple event ring segments coupling each event ringcircuit to the following event ring circuit. The ARM island 51 includesan event manager 251 and processor 253. A global event chain 247 couplesevent ring circuit 210 to event ring circuit 250 and event ring circuit250 to event manager 251 within ARM island 51. Operation of eventmanager 251 is disclosed below with respect to FIG. 42.

Event packets are communicated along first 245 and second 246 localevent rings as described above with respect to FIG. 40. With respect tothe first local event ring 245, once an event packet is communicated tothe second event ring circuit 210 within ingress NBI 72, the globalevent filter 227 within the second event ring circuit 210 monitors theevent packet and detects if the event packet represents a global event.If the global event filter 227 detects an event packet representing aglobal event, the event packet is copied to the global event chain 247.With respect to the second local event ring 246, once an event packet iscommunicated to the event ring circuit 250 within ME cluster island 54,a global event filter within event ring circuit 250 monitors the eventpacket and detects if the event packet represents a global event. If theglobal event filter within event ring circuit 250 detects an eventpacket representing a global event, the event packet is copied to theglobal event chain 247.

In one embodiment, the event ring circuit 248 within the ingress MEisland 66 inserts an event packet 281 into the first local event ring245. The event packet 281 represents a global event. The event packet281 is then communicated to the first event ring circuit 209 withiningress NBI island 72 via an event ring segment. Since an event packetgenerated by another event ring circuit is present, the first event ringcircuit 209 within ingress NBI island 72 cannot insert or delete anevent packet, but can monitor the contents of the event packet 281. Theevent packet 281 is then communicated to the second event ring circuit210 within ingress NBI island 72 via an event ring segment. Since anevent packet generated by another event ring circuit is present, thesecond event ring circuit 210 within ingress NBI island 72 cannot insertor delete an event packet, but can monitor the contents of the eventpacket 281. The global event filter 227 within event ring circuit 210determines that event packet 281 represents a global event. Upondetermining that the event packet 281 represents a global event, eventring circuit 210 will copy the event packet 281 to the first event chainsegment of the global event chain 247. The event packet 281 is alsocommunicated along the first local event ring 245 as described withrespect to FIG. 40 above. The event packet 281 then is communicatedalong a second event chain segment of the global event chain 247 toevent manager 251 within ARM island 51. Simultaneously, other eventpackets are communicated along the second local event ring 246. Theglobal event filter in event ring circuit 250 determines if eventpackets represent global events. Upon determining that an event packetrepresents a global event, the global event filter in event ring circuit250 copies the event packet to the second segment of the global eventchain 247. As shown above, the global event chain 247 allows eventpackets representing global events to be communicated to ARM island 51from both the first 245 and the second 246 local event rings.

It is noted that while select event packets are only copied to theglobal event chain 247 by global event filters; global event filters donot delete or “release” event packets from the first 245 or second 246local event rings. Only the event ring circuit which inserted the eventpacket into the local event ring can delete or “release” the said eventpacket from the local event ring. Event packets are only deleted fromthe global event chain by event manager 251 within ARM island 51. Globalevents may represent events such as error code correction (ECC) eventsor direct memory access (DMA) events.

In another embodiment, the event ring circuit 248 within the ingress MEisland 66 inserts the event packet 282 to the first local event ring245. Event packet 282 does not represent a global event. The eventpacket 282 is then communicated to the event ring circuit 209 withiningress NBI island 72 via an event ring segment. Since an event packetgenerated by another event ring circuit is present, event ring circuit209 within ingress NBI island 72 cannot insert or delete an eventpacket, but can monitor the contents of the event packet 282. Since theevent ring circuit 209 does not have a global event filter, the eventring circuit 209 will not determine if the event packet 282 represents aglobal event and will not copy the event packet 282 to the global eventchain. The event packet 282 is communicated to event ring circuit 210within ingress NBI island 72 via an event ring segment. Since an eventpacket generated by another event ring circuit is present, event ringcircuit 210 within ingress NBI island 72 cannot insert or delete anevent packet, but can monitor the contents of the event packet 282. Theglobal event filter 227 within event ring circuit 210 determines thatevent packet 282 does not represent a global event. Upon determiningthat the event packet 282 does not represent a global event, event ringcircuit 210 will not copy the event packet 282 to the first event chainsegment of the global event chain 247. The event packet 282 iscommunicated along the first local event ring 245 as described withrespect to FIG. 40 above. As shown above, the global event chain 247does not communicate event packets representing local event to the ARMisland 51.

FIG. 42 shows the event manager 251 of ARM island 51 in further detail.An event manager is also included in all cluster local scratch circuits,such as cluster local scratch 342 shown in FIG. 50. Event manager 251receives incoming global event packets from the last segment of theglobal event chain, detects various types of the global event packets,and collects and logs various types of information regarding thedetected types of global event packets. Event manager 251 may, forexample, count the number of certain types of global event packetsreceived. Event manager 251 may, for example, log certain informationabout certain types of global event packets. The collected data isstored in a data register. The ARM processor can configure the globalevent manager 251 via data bus 260 and set up which types of globalevent packets will be detected and logged. The ARM processor 253 canthen read the collected data from the data register via data bus 260.

In one embodiment, event manager 251 includes thirty-two event filters.One of the event filters 255 includes mask logic 262, a mask register256, match logic 263, a match register 257, configurable analysis logic261, a configuration register 258, and a data register 259. The lastsegment of the global event chain is coupled to an input of an amount ofmask logic 262. The mask logic 262 uses the value in mask register 256to mask out certain parts of the incoming global event packet. Theoutput of the mask logic 262 is supplied to match logic 263. The matchlogic 263 compares the output of the mask logic 262 to the value orvalues stored in match register 257. Match register 257 may, forexample, include an array of bit sequences. If the unmasked event packetbits output by the mask logic are determined to match one of the bitsequences stored in match register 257, then the match logic 263 outputsa digital value indicative of the occurrence. The digital value outputby the match logic is supplied to configurable analysis logic 261.Configurable analysis logic 261 may, for example, be configured by thevalue in configuration register 258 to be a counter so that theconfigurable analysis logic counts the number of matches that occur. Theoutput of the configurable analysis logic 261 is written into dataregister 259. Processor 253 can read the results from data register 259via data bus 260. Processor data bus 260 is coupled to a first input(CPU_DB) of processor 253.

Operational Example

FIG. 43 is a schematic diagram that illustrates an operational exampleof IB-NFP integrated circuit 12 within the MPLS router 1 of FIG. 1. 100Gbps packet traffic is received via optical cable 7 (see FIG. 1), flowsthrough optics transceiver 10, flows through PHY integrated circuit 11,and is received onto IB-NFP integrated circuit 12 spread across the fourSerDes I/O blocks 19-22. Twelve virtual input ports are provided at thisinterface in the example of FIG. 1. The symbols pass through directdedicated conductors from the SerDes blocks 19-22 to ingress MAC island71. Ingress MAC island 71 converts successive symbols delivered by thephysical coding layer into packets by mapping symbols to octets, byperforming packet framing, and then by buffering the resulting packetsfor subsequent communication to other processing circuitry. The packetsare communicated from MAC island 71 across a private inter-island bus toingress NBI (Network Bus Interface) island 72. Although dedicatedconnections are provided for this purpose in the particular exampledescribed here, in other examples the packets are communicated fromingress MAC island 71 to ingress NBI island via the configurable meshdata bus.

For each packet, the functional circuitry of ingress NBI island 72examines fields in the header portion to determine what storage strategyto use to place the packet into memory. In one example, the NBI islandexamines the header portion and from that determines whether the packetis an exception packet or whether the packet is a fast-path packet. Ifthe packet is an exception packet then the NBI island determines a firststorage strategy to be used to store the packet so that relativelyinvolved exception processing can be performed efficiently, whereas ifthe packet is a fast-path packet then the NBI island determines a secondstorage strategy to be used to store the packet for more efficienttransmission of the packet from the IB-NFP.

In the operational example of FIG. 43, NBI island 72 examines a packetheader, performs packet preclassification, determines that the packet isa fast-path packet, and determines that the header portion of the packetshould be placed into a CTM (Cluster Target Memory) in ME (Microengine)island 66. The header portion of the packet is therefore communicatedacross the configurable mesh data bus from NBI island 72 to ME island66. The CTM is tightly coupled to the ME. The ME island 66 determinesheader modification and queuing strategy for the packet based on thepacket flow (derived from packet header and contents) and the ME island66 informs a second NBI island 63 of these. In this simplified examplebeing described, the payload portions of fast-path packets are placedinto internal SRAM (Static Random Access Memory) MU block 78 and thepayload portions of exception packets are placed into external DRAM 40and 41.

Half island 68 is an interface island through which all informationpassing into, and out of, SRAM MU block 78 passes. The functionalcircuitry within half island 68 serves as the interface and controlcircuitry for the SRAM within block 78. For simplicity purposes in thediscussion below, both half island 68 and MU block 78 may be referred totogether as the MU island, although it is to be understood that MU block78 is actually not an island as the term is used here but rather is ablock. In one example, MU block 78 is an amount of so-called “IP” thatis designed and supplied commercially by a commercial entity other thanthe commercial entity that designs and lays out the IB-NFP integratedcircuit. The area occupied by block 78 is a keep out area for thedesigner of the IB-NFP in that the substantially all the wiring and allthe transistors in block 78 are laid out by the memory compiler and arepart of the SRAM. Accordingly, the mesh buses and associated crossbarswitches of the configurable mesh data bus, the mesh control bus, andthe mesh event bus do not pass into the area of block 78. No transistorsof the mesh buses are present in block 78. There is an interface portionof the SRAM circuitry of block 78 that is connected by short directmetal connections to circuitry in half island 68. The data bus, controlbus, and event bus structures pass into and over the half island 68, andthrough the half island couple to the interface circuitry in block 78.Accordingly, the payload portion of the incoming fast-path packet iscommunicated from NBI island 72, across the configurable mesh data busto SRAM control island 68, and from control island 68, to the interfacecircuitry in block 78, and to the internal SRAM circuitry of block 78.The internal SRAM of block 78 stores the payloads so that they can beaccessed for flow determination by the ME island.

In addition, a preclassifier in the ingress NBI island determines thatthe payload portions for others of the packets should be stored inexternal DRAM 40 and 41. For example, the payload portions for exceptionpackets are stored in external DRAM 40 and 41. Interface island 70, IPblock 79, and DDR PHY I/O blocks 46 and 47 serve as the interface andcontrol for external DRAM integrated circuits 40 and 41. The payloadportions of the exception packets are therefore communicated across theconfigurable mesh data bus from NBI island 72, to interface and controlisland 70, to external MU SRAM block 79, to 32-bit DDR PHY I/O blocks 46and 47, and to external DRAM integrated circuits 40 and 41. At thispoint in the operational example, the packet header portions and theirassociated payload portions are stored in different places. The payloadportions of fast-path packets are stored in internal SRAM in MU block78, whereas the payload portions of exception packets are stored inexternal SRAM in external DRAMs 40 and 41.

ME island 66 informs second NBI island 63 where the packet headers andthe packet payloads can be found and provides the second NBI island 63with an egress packet descriptor for each packet. The egress packetdescriptor indicates a queuing strategy to be used on the packet. SecondNBI island 63 uses the egress packet descriptor to read the packetheaders and any header modification from ME island 66 and to read thepacket payloads from either internal SRAM 78 or external DRAMs 40 and41. Second NBI island 63 places packet descriptors for packets to beoutput into the correct order. For each packet that is then scheduled tobe transmitted, the second NBI island uses the packet descriptor to readthe header portion and any header modification and the payload portionand to assemble the packet to be transmitted. Note that the headermodification is not actually part of the egress packet descriptor, butrather it is stored with the packet header by the ME when the packet ispresented to the NBI. The second NBI island then performs any indicatedpacket modification on the packet. The resulting modified packet thenpasses from second NBI island 63 and to egress MAC island 64.

Egress MAC island 64 buffers the packets, and converts them intosymbols. The symbols are then delivered by conductors from the MACisland 64 to the four SerDes I/O blocks 25-28. From SerDes I/O blocks25-28, the 100 Gbps outgoing packet flow passes out of the IB-NFPintegrated circuit 12 and across SerDes connections 34 (see FIG. 1) andto switch fabric 9. Twelve virtual output ports are provided in theexample of FIG. 1.

FIG. 44 is a diagram of the four SerDes I/O blocks 19-22 and the ingressMAC island 71. The symbols 300 pass from the four SerDes I/O blocks andto the ingress MAC island across dedicated conductors 301. The symbolsare converted into packets by a 100 Gbps ethernet block 302. The 100Gbps ethernet block 302 analyzes the packets and places the results inthis analysis at the beginning of the packet in the form of a “MACprepend” value. The resulting packets and associated MAC prepend valuesare then buffered in SRAM 305. Reference numeral 303 identifies a partof the block that represents one packet and reference numeral 304identifies a part of the block that represents the MAC prepend value.The MAC prepend value 304 includes: 1) an indication of the length ofthe packet, 2) an indication whether the packet is an IP packet, 3) andindication of whether the checksums are correct, and 4) a time stampindicating when the packet was received.

As packets are loaded into SRAM, a statistics block 306 counts thenumber of packets that meet certain criteria. Various sub-circuits ofthe ingress MAC island are configurable. The input conductors 307labeled CB couples the certain portions of the MAC island to the controlbus tree illustrated in FIG. 29 so that these portions receiveconfiguration information from the root of control bus tree. SRAM block305 includes error detection and correction circuitry (ECC) 308. Errorinformation detected and collected by ECC block 308 and statistics block306 is reported through the local event bus and global event chain backto the ARM island 51 by the mechanism described above in connection withFIG. 29. Ingress MAC island 71 is part of one of the local event rings.Event packets are circulated into the MAC island via conductors 309 andare circulated out of the MAC island via conductors 310. Packets thatare buffered in SRAM 305 are then output from the MAC island to theingress NBI island 72 in the form of one or more 256 byte minipackets311 communicated across dedicated connections 312. Statisticsinformation 313 is also communicated to the ingress NBI island 72 viadedicated connections 314.

FIG. 45 is a diagram of packet 303 communicated across connections 312.

FIG. 46 is a diagram of ingress NBI island 72. Ingress NBI island 72receives the MAC prepend and the minipackets via dedicated connections312 from the ingress MAC island 72. The first 256 bytes of the packetand the MAC prepend pass through multiplexing circuitry 315 and to acharacterizer 316. Characterizer 316 outputs characterizationinformation, the first sixty-four bytes of the packet, and the MACprepend. This is passed to a pool 317 of forty-eight picoengines. Eachpicoengine executes a program stored in an associated instructioncontrol store. Reference numeral 318 identifies the first picoengine andreference numeral 319 identifies its instruction control store. Theprogram in the instruction control store for each picoengine can beupdated and changed under software control via control block 320.Control block 320 is also usable to receive the statistics information313 from the MAC island via XPB bus connections 314. To perform deeperand deeper analysis into the header structure of an incoming packet, theoutput of the pool 317 can be passed back through a tunnel recirculationpath and tunnel recirculation FIFO 400 to the characterizer 316 in aniterative fashion. Pool 317 outputs preclassification results 321.

FIG. 47 is a table that sets forth the part of preclassification results321. The preclassification results 321 include: 1) a determination ofwhich one of multiple buffer pools to use to store the packet, 2) asequence number for the packet in a particular flow of packets throughthe IB-NFP, and 3) user metadata. The user metadata is typically a codegenerated by the picoengines, where the code communicates certaininformation about the packet. In the present operational example, theuser metadata includes a bit. If the bit is set then the packet wasdetermined to be of a first type (an exception packet), whereas if thebit is not set then the packet was determined to be of a second type (afast-path packet).

The packet is buffered in SRAM 322. A buffer pool is a set of targets inME islands where header portions can be placed. A buffer list is a listof memory addresses where payload portions can be placed. DMA engine 323can read the packet out of SRAM via conductors 324, then use the bufferpools to determine a destination to which the packet header is to be DMAtransferred, and use the buffer lists to determine a destination towhich the packet payload is to be DMA transferred. The DMA transfersoccur across the configurable mesh data bus. In the case of theexception packet of this example the preclassification user metadata andbuffer pool number indicate to the DMA engine that the packet is anexception packet and this causes a first buffer pool and a firstdifferent buffer list to be used, whereas in the case of the fast-pathpacket the preclassification user metadata and buffer pool numberindicate to the DMA engine that the packet is a fast-path packet andthis causes a second buffer pool and a second buffer list to be used.Block 326 is data bus interface circuitry through which the configurablemesh data bus in accessed. Arrow 325 represents packets that are DMAtransferred out of the NBI island 72 by DMA engine 323. Each packet isoutput with a corresponding ingress packet descriptor.

FIG. 48 is a table that sets forth the parts of an ingress packetdescriptor. An ingress packet descriptor includes: 1) an addressindicating where and in which ME island the header portion is stored, 2)an address indicating where and in which MU island the payload portionis, 3) how long the packet is, 4) a sequence number for the flow towhich the packet belongs, 5) user metadata.

The programs stored in the instruction stores that are executable by thepicoengines can be changed multiple times a second as the routeroperates. Configuration block 327 receives configuration informationfrom the control bus CB tree via connections 328 and supplies theconfiguration information to various ones of the sub-circuits of NBIisland 72 that are configurable. Error detection and correction (ECC)circuitry 329 collects error information such as errors detected in thecontents of the instruction stores. ECC circuitry 329 and ECC circuitry330 are coupled via connections 331 and 332 and other internal islandconnections not shown to be part of the local event ring of which theingress MAC island 72 is a part.

FIG. 49 is a diagram of the microengine (ME) island 66. In the presentoperational example, packet headers and the associated preclassificationresults are DMA transferred from the ingress NBI island 72 across theconfigurable mesh data bus and into the Cluster Target Memory (CTM) 333of the ME island 66. The DMA engine 323 in the ingress NBI island is themaster and the CTM 333 is the target for this transfer. The packetheader portions and the associated ingress packet descriptors pass intothe ME island via data bus island bridge 334 and data bus interfacecircuitry 335. Once in the CTM 333, the header portions are analyzed byone or more microengines. The microengines have, through the DB islandbridge 334, a command out interface, a pull-id in interface, a pull-dataout interface, and a push data in interface. There are six pairs ofmicroengines, with each pair sharing a memory containing program codefor the microengines. Reference numerals 336 and 337 identify the firstpair of picoengines and reference numeral 338 identifies the sharedmemory. As a result of analysis and processing, the microengines modifyeach ingress packet descriptor to be an egress packet descriptor. Eachegress packet descriptor includes: 1) an address indicating where and inwhich ME island the header portion is found, 2) an address indicatingwhere and in which MU island the payload portion is found, 3) how longthe packet is, 4) sequence number of the packet in the flow, 5) anindication of which queue the packet belongs to (result of the packetpolicy), 6) an indication of where the packet is to be sent (a result ofthe packet policy), 7) user metadata indicating what kind of packet itis.

Memory errors and other events detected in the ME island are reportedvia a local event ring and the global event chain back to the ARM island51. A local event ring is made to snake through the ME island for thispurpose. Event packets from the local event chain are received viaconnections 339 and event packets are supplied out to the local eventchain via connections 340. The CB island bridge 341, the cluster localscratch 342, and CTM 333 can be configured and are therefore coupled tothe control bus CB via connections 343 so that they can receiveconfiguration information from the control bus CB.

A microengine within the ME island can use data bus commands to interactwith a target, regardless of whether the target is located locally onthe same ME island as the microengine or whether the target is locatedremotely in another island, using the same configurable data buscommunications. If the target is local within the ME island, then themicroengine uses data bus commands and operations as described above asif the memory were outside the island in another island, except that bustransaction values do not have a final destination value. The bustransaction values do not leave the ME island and therefore do not needthe final destination information. If, on the other hand, the target isnot local within the ME island then intelligence 343 within the DBisland bridge adds the final destination value before the bustransaction value is sent out onto the configurable mesh data bus. Fromthe perspective of the microengine master, the interaction with thetarget has the same protocol and command and data format regardless ofwhether the target is local or remote.

FIG. 51 is a diagram of a bus transaction value 344 used to communicatepacket data from the ingress NBI island 72 to the ME island 66. In amulti-target island such as the ME island 66, the target field 345 ofthe bus transaction value contains a number that indicates which targetit is that is to receive the payload of the bus transaction value. Inthe present example, the header portions of the incoming 100 Gbps floware written into CTM 333.

The local event ring flow through ME island 66 illustrated in FIG. 50.The local event ring couples from the event bus (EB) through CB islandbridge 341, cluster local scratch 342, cluster target memory 333 andback to event bus (EB). Local and global events are communicated betweenCB island bridge 341, cluster local scratch 342, cluster target memory333 via the local event bus. Each cluster local scratch 342 contains anevent manager 251 as illustrated in FIG. 42.

FIG. 52 is a diagram of MU half island 68 and SRAM block 78. MU halfisland 68 includes several hardware engines 350. In the operationalexample, packet payloads are DMA transferred directly from ingress NBIisland 72 and across the configurable mesh data bus, through data businterface 352 of half island 68, and into the data cache SRAM 351 block78. The ingress NBI DMA engine 323 issues a bulk write command acrossthe configurable mesh data bus to the bulk transfer engine 346. Thedestination is the MU island. The action is bulk write. The addresswhere the data is to be written into the MU island is the address takenout of the appropriate buffer list. The bulk write command received atthe MU is a bulk write, so the data bus interface 352 presents thecommand to the bulk engine. The bulk engine examines the command whichis a write. In order to perform a write the bulk engine needs data, sothe bulk engine issues a pull-id through the pull portion of interface352, which in turn issues a pull-id back onto the configurable mesh databus. The NBI DMA engine 323 receives the pull-id. Part of the pull-id isa data reference which indicates to the DMA engine which part of thepacket is being requested as data. The DMA engine uses the datareference to read the requested part of the packet, and presents thatacross the data part of the data bus back to the bulk engine 346. Thebulk engine 346 then has the write command and the packet data. The bulkengine 346 ties the two together, and it then writes the packet datainto the SRAM 351 at the address given in the write command. In thisway, packet payload portions pass from DMA engine in the ingress NBIisland, across the configurable mesh data bus, through the data businterface 352, through a bulk transfer engine 346, and into data cacheSRAM 351.

In the present operational example, a microengine in the ME island 66issues a lookup command across the configurable mesh data bus to havelookup hardware engine 350 examine tables in SRAM 351 for the presenceof given data. The data to be looked for in this case is a particularMPLS label. The lookup command as received onto the MU island is alookup command so the data base interface 352 presents the lookupcommand to the lookup engine. The lookup command includes a tabledescriptor of what part to memory to look in. The lookup command alsocontains a pull-id reference indicating what to look for (the MPLS labelin this case). The data to look for is actually stored in transferregisters of the originating microengine. The lookup engine 350therefore issues a pull-id out onto the configurable mesh data busrequest back to the originating microengine. The microengine returns therequested data (the MPLS label to look for) corresponding to thereference id. The lookup engine now has the lookup command, the tabledescriptor, and the MPLS label that it is to look for. In theillustration there are three tables 353-355. A table descriptionidentifies one such table by indicating the starting address of thetable in SRAM 351, and how large the table is. If the lookup operationis successful in that the lookup hardware engine 350 finds the MPLSlabel in the table identified by the table descriptor, then the lookuphardware engine 350 returns a predetermined value “Packet Policy” 356back to the requesting microengine. A packet policy is a code thatindicates: 1) a header modification to be done, and 2) a queueingstrategy to use. Lookup engine 350 returns the packet policy 356 to theoriginating microengine by pushing the data (the packet policy) via thepush interface of the configurable mesh data bus.

Various parts of the MU island are configurable by changing the contentsof registers and memory via the control bus CB and connections 357 andcontrol status registers 362. Errors detected on the MU island bycircuits 360 and 361 are reported into a local event ring. Event packetsfrom the local event ring are received via input connections 358 and theMU island outputs event packets to the local even ring via outputconnections 359. Various sub-circuits of the MU island are configurable.

FIG. 53 is a diagram of egress NBI island 63. In the operationalexample, ME island 66 instructs the egress NBI island 63 to transmit apacket by supplying the egress NBI island with an egress packetdescriptor of the packet to be transmitted. The ME island supplies theegress packet descriptor to the egress NBI island by issuing a transmitpacket command across the configurable mesh data bus and to the packetreorder block 401. The packet reorder block 401 responds by pulling thepacket descriptor from the ME island across the configurable mesh databus. In this way, multiple egress packet descriptors enter packetreorder block 401. These egress packet descriptors are reordered so thatthe descriptors for the packets of a flow are in proper sequence. Thescheduler 366 receives the properly ordered egress packet descriptorsand pushes them onto appropriate queues in queue SRAM 367. Each suchqueue of egress packet descriptors is per port, per data type, per groupof connections. Reference numeral 368 identifies one such queue. Packetsof a connection in this case share the same set of source anddestination IP addresses and TCP ports. Scheduler 366 schedules packetsto be transmitted by popping egress packet descriptors off the queues inappropriate orders and at appropriate times, and by supplying the poppedegress packet descriptors via conductors 381 to the DMA engine 363.

DMA engine 363 receives such an egress packet descriptor, and based onthe information in the descriptor, transfers the payload portion and theheader portion of the packet across configurable mesh data bus and DBinterface 364 and into FIFO 365. In the illustration of FIG. 47, eachentry in FIFO 365 includes a complete packet having the header portion371, the payload portion 372, and a script identifier portion 373. Thescript identifier portion 373 was added by the ME island. As a result ofthe lookup performed at the direction of the ME island, a packet policywas determined, and part of this packet policy is an indication of whatof the packet header to change and how to change it before the packet istransmitted. An example of such a modification is to change the MACsource and destination addresses at the time the packet is output fromthe IB-NFP.

In a typical MPLS router, the MPLS labels of packets can remain the sameas the packets flow into and through and out of the router. The MACaddresses of such a packet, however, should be changed on a hop by hopbasis. The MAC hop on the ingress may be different from the MAC addresson the egress. Accordingly, the packet exiting the MPLS router shouldhave its source and destination MAC addresses changed to be appropriatefor the next MAC hop into which the packet will be transmitted. The MEisland supplies a script identifier portion for each packet for thispurpose. The script identifier portion includes a code that identifiesone of the scripts present in script SRAM 375. The identified script,when executed by packet modifier 374, causes the MAC addresses for theassociated packet to be changed to values stored in an associatedargument SRAM 376. Each resulting modified packet is then output fromthe egress NBI island 63 as a sequence of 256 byte minipackets acrossdedicated connections 369 to egress MAC island 64. Reference numeral 370identifies one such minipacket.

Error conditions detected by ECC circuits 377 and 378 are injected intoa local event ring in the form of event packets. Event packets from thelocal event ring are received onto the egress NBI island via connections379, and event packets from the egress NBI island are supplied throughthe remainder of the local event ring via connections 380. Various partsof the egress NBI island are configurable. Configuration information forthis purpose is received onto the egress NBI island from the control busCB via connections 382.

FIG. 54 is a diagram of egress MAC island 64. A packet 383 fortransmission are received from egress NBI island 63 in the form ofminipackets 370 via dedicated connections 369. The packets are bufferedin SRAM 384. In the operational example, the packets to be output fromthe egress MAC island via are converted into symbols by Interlaken block385. The resulting symbols 386 pass via dedicated connections 387 to thefour SerDes I/O blocks 25-28. As described above in connection with FIG.1, the four SerDes I/O blocks are coupled by SerDes connections 29 toswitch fabric 9 of the MPLS router 1. ECC circuitry 388 of SRAM 384 ismade a part of a local event ring via EB connections 389 and 390.Sub-circuits of MAC island are configurable. Configuration informationfor these sub-circuits is received from the control bus tree viaconnections 391.

FIG. 55 is a diagram that illustrates a packet flow in the operationalexample when local memory resources in the CTM 333 of the ME island 66are determined not to be scarce. An incoming packet passes through theingress MAC island 71 and the ingress NBI island 72 as described above.Arrow 392 indicates that the header portion is then transferred (Step 1)across the configurable mesh data bus into CTM 333 of ME island 66,whereas arrow 393 indicates that the payload portion of the packet istransferred (Step 2) across the configurable mesh data bus into the MUisland 68,78 without being stored in the ME island. The payload portionof each packet is stored in the MU island such that spare memory spaceis left at the beginning of where the payload is stored. That sparememory space is adequate to accommodate the header portion of the packetwithout overwriting other packet payloads. In the case of FIG. 55,however, the header portion is never written into the MU island.Microengines of the ME island and hardware engines of the MU islandanalyze and process the packet. Arrow 394 indicates that the headerportion is then transferred (Step 3) from the ME island 66 and to theegress NBI island 63. Arrow 395 indicates that the payload portion istransferred (Step 3) from the MU island 68, 78 to the egress NBI island63. The same step number is used because these transfers may occursimultaneously. The header portion and the payload portion are combinedin the NBI island 63 and then pass through the egress MAC island 64 andthe four SerDes I/O blocks and out of the IB-NFP integrated circuit.

FIG. 56 is a diagram that illustrates a packet flow in the operationalexample when local memory resources in the CTM 333 of the ME island 66are determined to be scarce. An incoming packet passes through theingress MAC island 71 and the ingress NBI island 72 as described above.As indicated by arrow 396, the header portion is then transferred (Step1) across the configurable mesh data bus into CTM 333 of ME island 66.Arrow 396 indicates that the payload portion of the packet istransferred (Step 2) across the configurable mesh data bus into the MUisland 68, 78 without being stored in the ME island. As in the casedescribed above in connection with FIG. 55, the payload portion of eachpacket is stored in the MU such that spare memory space exists at thebeginning of where the payload is stored. The spare memory space isadequate to accommodate the header portion without overwriting otherpacket payloads. Based on how long it will take before the packet willbe transmitted from the IB-NFP, the egress NBI island 63 determines thatthe header portion shall be moved from the ME island and to MU island inorder to free up resources in the CTM 333 of the ME island. As indicatedby arrow 398, the header portion is transferred (Step 3) from the MEisland and is stored into the ME island into the spare memory space atthe beginning of its associated payload portion. Microengines of the MEisland and hardware engines of the MU island analyze and process thepacket. The packet may be analyzed before the transfer 398, or after thetransfer 398, or both. When the scheduler of the egress NBI islanddetermines that the packet is to be transferred for transmission fromthe IB-NFP integrated circuit, then the header portion and the payloadportion are DMA transferred (Step 4) together directly from the MUisland and across the configurable mesh data bus and to the egress NBIisland. Arrow 399 indicates this transfer of the packet header and thepacket payload. The packet then passes across dedicated connections fromthe egress NBI island 63 to the egress MAC island 64, and through thefour SerDes blocks, and out of the IB-NFP integrated circuit.

FIG. 57 is a configurable mesh event bus configured to take correctiveaction when system resources within an island are determined to bescarce. FIG. 57 shows the configurable mesh event bus of FIG. 39configured to form a first 245 and a second 246 local event ring and asingle global event chain 247. NBI island 72 corresponds to the NBIisland 72 shown in FIGS. 33 and 34. The first local event ring 245 flowsthrough the ingress MAC island 71, ingress NBI island 72, ingress MEisland 66, ARM island 51 and eight other islands (53, 57, 62, 65, 60,55, 61, and 56). The second local event ring 246 flows through theegress NBI island 63, egress MAC island 64 and four other islands (54,58, 59, and 69). The global event chain 247 begins on ingress NBI island72, couples through island 66, island 62, island 57, island 54 andterminates on ARM island 51. Both local event rings are unidirectional,in that event packets travel in a single direction along the local eventring. The global event chain is unidirectional, in that global eventpackets travel in a single direction along the global event chain. Whilethe global even chain has a single point of termination on ARM island51, both local event rings have no fixed termination location. Rather,event packets are removed at different points along the local event ringdepending upon where in the local event ring the event packet wasinserted into the local event ring. Each local event ring operates as asource-release ring.

An exemplary request for additional system resources communicated acrossvarious islands is shown in FIG. 57. Ingress NBI island 72 determines(Step 1) that additional system resources are required. In response tothe need for additional system resources, ingress NBI island 72 insertsan event packet into first local event ring 245. The event packetindicates the source of the event packet and the system resourcerequested. The event packet is communicated along the first local eventring 245 to ME island 66. Event ring circuit 248 within ME island 66receives (Step 2) the event packet from ingress NBI island 72. Inresponse to receiving the event packet requesting additional systemresources, ME island 66 communicates a request for additional resourcesacross the configurable mesh data bus to egress NBI island 63 (Step 3).In step 4, egress NBI island 63 determines what system resources areavailable and allocates the required system resources for use by ingressNBI 72. After allocating the required system resources, egress NBIisland 63 communicates information regarding the allocated resourcesacross the configurable mesh data bus to ingress NBI island 72 (Step 5).In step 6, ingress NBI island 72 receives the information regarding theallocated resources and begins utilization of the requested systemresources. In an embodiment, a system resource is a processing resource.A processing resource may be an amount of buffer memory. A buffer memorymay be used for storing incoming packet information. Additionally,information regarding the allocated resources may be a pointer addressfor the buffer memory being allocated.

Although certain specific embodiments are described above forinstructional purposes, the teachings of this patent document havegeneral applicability and are not limited to the specific embodimentsdescribed above. Accordingly, various modifications, adaptations, andcombinations of various features of the described embodiments can bepracticed without departing from the scope of the invention as set forthin the claims.

What is claimed is:
 1. A method comprising: (a) receiving a first packetonto an island-based network flow processor (IB-NFP) integrated circuit,wherein the IB-NFP integrated circuit comprises a first island, a secondisland, a third island, and a fourth island, and wherein the firstpacket comprises a first header portion and a first payload portion; (b)communicating the first header portion from the first island to thesecond island and storing the first header portion in a local memory onthe second island; (c) communicating the first payload portion from thefirst island to the third island and storing the first payload portionin a remote memory; (d) communicating the first header portion from thefirst island to the fourth island without the first header portionpassing through the third island; (e) communicating the first payloadportion from the third island to the fourth island without the firstpayload portion passing through the second island; (f) outputting thefirst packet from the IB-NFP integrated circuit; (g) receiving a secondpacket onto the IB-NFP integrated circuit, wherein the second packetcomprises a second header portion and a second payload portion; (h)communicating the second header portion from the first island to thesecond island and storing the second header portion in the local memoryon the second island; (i) communicating the second payload portion fromthe first island to the third island and storing the second payloadportion in the remote memory; (j) communicating the second headerportion from the second island to the third island and storing thesecond header in the remote memory; (k) communicating both the secondheader portion and the second payload portion from the third island tothe fourth island; and (l) outputting the second packet from the IB-NFPintegrated circuit.
 2. The method of claim 1, wherein the remote memoryis a part of the third island.
 3. The method of claim 1, wherein theremote memory is a memory device external to the IB-NFP integratedcircuit.
 4. The method of claim 1, wherein the second island has anamount of a processing resource, the method further comprising:determining that a packet header stored in the local memory in thesecond island will be communicated from the second island to the thirdisland based at least in part on the amount of the processing resource.5. The method of claim 4, wherein the amount of the processing resourceis an amount of available memory space in the local memory of the secondisland.
 6. The method of claim 4, further comprising: determining that apacket header stored in the local memory in the second island will becommunicated from the second island to the third island based at leastin part on scheduling information.
 7. The method of claim 6, where thescheduling information is generated in the fourth island.
 8. The methodof claim 1, wherein the second payload portion is stored in the remotememory in (i) such that an amount of unused memory space existsimmediately before memory space used to store the second payloadportion, and wherein the header portion is stored in the remote memoryin (j) in the unused memory space.
 9. The method of claim 1, wherein thesecond header is stored in the remote memory in (j) such that the secondheader and the second payload occupy one contiguous memory space in theremote memory.
 10. The method of claim 1, wherein the local memory onthe second island is a Cluster Target Memory (CTM), and wherein thesecond header portion is communicated in (j) from the second island tothe third island via a configurable mesh data bus.
 11. A methodcomprising: (a) receiving a packet onto an integrated circuit andstoring a header portion of the packet in a memory in a first island ofthe integrated circuit and storing a payload portion of the packet inanother memory; (b) determining to communicate the header portion fromthe memory in the first island to the other memory based at least inpart on an amount of a processing resource; (c) in response to thedetermining of (b) communicating the header portion from the memory inthe first island to the other memory; and (d) communicating both theheader portion and the payload portion from the other memory to a secondisland, wherein the second island comprises a memory into which theheader portion and the payload portion are stored, wherein the secondisland further comprises an egress scheduling circuit.
 12. The method ofclaim 11, wherein the amount of the processing resource is an amount ofavailable memory space in the memory in the first island.
 13. The methodof claim 11, wherein the header portion is communicated in (d) from theother memory to the second island via a configurable mesh data bus onthe integrated circuit.
 14. The method of claim 11, wherein a flow ofpackets is received onto the integrated circuit and is stored in thefirst island, wherein header portions of packet of the flow of packetsare not stored in the other memory but rather are stored in the memoryin the first island and are then communicated from the first island tothe second island without being stored in the other memory as long asthe amount of the processing resource remains above a predeterminedthreshold amount, whereas if the amount of the processing resource dropsbelow the predetermined threshold amount then header portions of theflow of packets begin to be buffered into the other memory before beingcommunicated to the second island.
 15. The method of claim 14, whereinthe other memory is a block of memory on the integrated circuit, andwherein the block is larger than either the first island or the secondisland.
 16. The method of claim 14, wherein the other memory is a memoryexternal to the integrated circuit.
 17. An integrated circuitcomprising: a first island that receives a plurality of packets, whereineach packet includes a header portion and a payload portion; a secondisland to which the header portions are transferred from the firstisland; a third island to which the payload portions are transferredfrom the first island; a fourth island that receives the plurality ofpackets, wherein the fourth island receives each of the packets byreceiving the payload portion of said each packet from the third islandand either receiving the payload portion of said each packet from eitherthe second island or from the third island; and means for determiningthat the header portions of some packets should be transferred from thesecond island to the third island before being transferred from thethird island to the fourth island and also for determining that theheader portions of other packets should not be transferred from thesecond island to the third island, wherein the determining is based atleast in part on an amount of a processing resource.
 18. The integratedcircuit of claim 17, wherein the means is a processor of the secondisland.
 19. The integrated circuit of claim 17, wherein the amount ofthe processing resource is an amount of available memory space in thesecond island.
 20. The integrated circuit of claim 17, wherein the thirdisland includes an interface for a memory external to the integratedcircuit, wherein the payload portions are stored in the external memory.